Will Hopkins is an independent blog covering personal and indieweb & blogging. It publishes a few times per month, with 1 reader following along on Blogs Are Back.
Occasional
Publishes a few times per month
1
Independent Blog
English
How this blog's content is accessed through Blogs Are Back.
Full Content
RSS feed includes complete post content for reading in-app
Proxy Required
Feed is fetched through our proxy for browser compatibility
Proxy Post Links
Post pages are loaded through our proxy for compatibility
Embeddable
Posts can be displayed inline in the reader view
Recent posts from Will Hopkins's RSS feed.
The point of data engineering
The number one thing that keeps getting hammered home to me, by experience and the folks at work I'm responsible to, is that the point of data/analytics engineering is to generate value for the business.New pipelines and data sources don't inherently create any value. Usually, they're a net loss. Doing your own job well isn't enough: you need to make the data easy to access and use, and deliver it to the right people at the right time. Every data engineer should be at least a...
Replicating Postgres
I finally finished my albatross project: replicating all our Postgres shards to Databricks. This is great news.In broad strokes, we export our Postgres databases daily from the backup snapshot, then ingest those Parquet snapshots with Databricks. Previously, we had done a hacked-together variant of the same process using only AWS tools, but it was a bit tougher to stitch things together. While AWS tends to take a bit of a LEGO approach (a bunch of building blocks you can recombine into bigger st...
My highly effective habit for feeling better
Every morning I start the day by making coffee, feeding the cats, heating up a hot breakfast for myself, and then I sit with my coffee, a lap cat, and my happy light for 30-60 minutes. This trio of activities (coffee, food, light) happens as soon as I can after waking up, every single day of the dark months.Please note that the title is a bit sarcastic, calling back to 7 Habits of Highly Effective People. That said, it really has changed my life. Along with a daily B12 and D3 vitamin supplement,...
Sunday upgrades
It's Sunday, and I was able to get some downtime at work to do some upgrades behind the scenes. I've been working on this data pipeline for months, and today I finally got to enable the first bit of it in production. We still have quite a bit of work left to do, but this is a step in the right direction!A few takeaways from the process so far:Don't take LLMs at their word, whether they're yours or a coworkers. Verify everything they do.It's worth waiting to commit your...
100 posts, and a year-ish
I've been writing here for about a year and this is my 100th post. It's been quite a year for all of us, that much is certain.I went to IrelandI have been at my current employer for longer than I've been at any startup (though a fraction of the time I spent at a corporate job)I traveled to Boston at least 6 times to visit family and workI hosted family in Ann Arbor twiceI wrote and thought a great deal about politics, much more than I'd have likedI cooked some food, though le...
Follow Will Hopkins
Add this blog to your reading list on Blogs Are Back, or visit the blog directly.