6 minutes ago by heliodor
If we consider Graphite, InfluxDB, and Prometheus, at this point in the monitoring industry's evolution, we have the capability to easily criss-cross metrics generated in the format of one of these systems to store them in one of the other ones.
The missing piece remains to be able to query one system with the query language of the others. For example, query Prometheus using Graphite's query language.
10 hours ago by bitcharmer
You'd be surprised how many serious tech shops have close to zero performance metrics collected and utilised.
I've done this in fintech a few times already and the best stack that worked from my experience was telegraf + influxdb + grafana. There are many things you can get wrong with metrics collection (what you collect, how, units, how the data is stored, aggregated and eventually presented) and I learned about most of them the hard way.
However, when done right and covering all layers of your software execution stack this can be a game changer, both in terms of capacity planing/picking low hanging perf fruit and day to day operations.
Highly recommend giving it a try as the tools are free, mature and cover a wide spectrum of platforms and services.
an hour ago by chucky_z
I'd like to followup on this and say InfluxDB is fantastic -- with the caveat that you know exactly what you want, and use it with that in mind. It's great at storing some metrics with high volume, but I've found the OSS version falls over hard in some circumstances (high cardinality in tags absolutely murders performance, making it nearly unusable).
I'm in the middle of doing a test-run of vmagent + victoria-metrics as a mostly-replacement (see: not replacing the cases where it's known exactly what's wanted, and used explicitly for that), and victoria-metrics fully support telegraf pushes which is a big bonus.
Also they're apparently going to be dropping InfluxQL entirely, for Flux, which is weird to me? The docs don't outright state this, but InfluxDB 2.0 has only Flux examples in the docs I can find. It's a better language, however the learning curve is not trivial.
an hour ago by duckmysick
> There are many things you can get wrong with metrics collection
Can you share the important things that are overlooked?
9 hours ago by benraskin92
Might also want to give Prometheus a try as it's extremely simple to setup, and it is very well supported in the open source community.
8 hours ago by bitcharmer
Prometheus was another option but millisecond-precise timestamps are a deal breaker in my field.
7 hours ago by benraskin92
That makes sense -- have you taken a look at M3DB (https://www.m3db.io/)?
8 hours ago by amenod
Curious, would microseconds suffice? Or are we talking about higher precision still?
9 hours ago by chrisweekly
Agreed; same exp (w many years of web perf optimization as ui arch and/or perf strategy consulting gigs. Even after all these years, simply using WebPageTest to capture test runs and share the results is often a transformational experience.
17 minutes ago by sa46
Speaking of high cardinality metrics, what are good options that aren’t as custom as map reduce jobs and a bit more real time?
We killed our influx cluster numerous times with high cardinality metrics. We migrated to Datadog which charges based on cardinality so we actively avoid useful tags that have too much cardinality. I’m investigating Timescale since our data isn’t that big and btrees are unaffected by cardinality.
9 hours ago by jamessun
"I don't have anything against hot new technologies, but a lot of useful work comes from plugging boring technologies together and doing the obvious thing."
9 hours ago by Simulacra
The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” (I found it!) but “That’s funny …” — Isaac Asimov
5 hours ago by m463
But that's not as fun as doing a scheme to rust cross-compiler on kubernetes
9 hours ago by dirtydroog
wise words
8 hours ago by resu_nimda
Starting the article off with "I did this in one day" - complete with a massive footnote disclaiming that it obviously took a lot more than one day - kinda ruined it for me. Why even bother with that totally unnecessary claim?
7 hours ago by eshyong
My read on it is the author is saying that seemingly small changes can have big impacts. I agree it could have been worded better, though I doubt he's trying to promote himself as a genius (as other people are saying) because he clearly highlights the effort his team put into the project in the footnote.
8 hours ago by caiobegotti
It's kind of a personal marketing thing these days to have this maverick/hero aura of genius instead of the "unproductive" but real and hard grinding work to get something done and delivered. It worked for a few so thousands try the same and we are here now, I guess?
7 hours ago by derivativethrow
Given:
- the context that the author already has a very successful career as a well-known developer
- the humility he evidences in most posts on his blog
- the fact that he explicitly highlights the work of others in this post alongside his own
I really don't think Dan is doing this as any form of personal marketing. He has no need of personal marketing, his blog already has several million views per month and frequently shows up on HN as it is, and it isn't really his style.
6 hours ago by caiobegotti
I did not say he did that, I said that I believe it's common these days given the points I mentioned. You just need to hang around and see a bunch of posts on HN to notice that. QED, he's probably one of the "few" I talked about.
2 hours ago by jacques_chester
In fairness, based on the limited time I've spent in his company, Dan Luu is pretty bright.
7 hours ago by derivativethrow
That strikes me as a short tolerance for feeling something is ruined. He appropriately highlighted the real time estimate of the more involved work in a footnote. He didn't literally mean all of the work was one day, he's trying to convey a larger point about outsized engineering returns from comparatively small person-hours of work.
Were you able to move past this to read the rest of the article? Because it's a very good article.
6 hours ago by resu_nimda
I did go back and read the rest of it, and I do agree that it's pretty good overall. For someone going into detail about the mistaken capitalization of a variable name (which I appreciated), the "one day" bit still stands out as oddly hand-wavy and boastful (and, as the opening remark, I would argue it's pretty important for setting the tone).
If that point needed to be made (and I don't think it really did in this article, that's not the focus), it could have been done more carefully.
7 hours ago by brmgb
I was really off-put by it too.
"I did it by myself in one day, well actually it was one week but had I known the stack I would likely have done it in one day. Oh, and by the way, after that week, there was yet another month of work involving at least two other persons from my team and then even more work from other teams. But let's not dwell on boring details".
It's nearly as infuriating as the "Appendix: stuff I screwed up" which doesn't contain actual screw up. It's a shame because the rest of the writing is interesting and doesn't need to be propped up.
3 hours ago by staysaasy
The boring technology observation (here referring to the challenge of getting publicity for "solving a 'boring' problem") is really true.
It extends very well to something that we constantly hammer home on my team: using boring tools is often best because it's easier to manage around known problems than forge into the unknown, especially for use-cases that don't have to do with your core business. Extreme & contrived example: it's much better to build your web backend in PHP over Rust because you're standing on the shoulders of decades of prior work, although people will definitely make fun of you at your next webdev meetup.
(Functionality that is core to your business is where you should differentiate and push the boundaries on interesting technology e.g. Search for Google, streaming infrastructure for Netflix. All bets are off here and this is where to reinvent the wheel if you must – this is where you earn your paycheck!)
10 hours ago by gigatexal
This is a really awesome blog. The post about programmer salaries is insightful: https://danluu.com/bimodal-compensation/
8 hours ago by julianeon
I was thinking the answer to 'why are programmers paid more than other white-collar professions that are similarly profitable' is: because programmers control the means of production.
I might be a great telecomm tech, a genius even, but once I'm out of a job, I can't build my own telecomm system - that would cost billions. I have to go back to some other telecomm system to start making money again.
But, at a startup, a kicked-out senior engineer can actually pretty much exactly recreate the company; they can do the equivalent of a laid-off telecomm employee starting a new, almost-as-good (except for branding) telecomm company.
No billions in infrastructure required: within a month or two, the cloned company could be near-indistinguishable from the original.
So companies have to pay employees more like partners, instead of employees, because either they pay them as equals or they'll be forced to compete against them, as equal rivals.
7 hours ago by SpicyLemonZest
Interesting. I hadn't thought about it that way before, but it does seem to predict the bimodality; the lower mode is (presumably) programmers who either aren't skilled enough or don't work in the right areas to be able to take their ball and found a startup with it.
9 hours ago by undefined
9 hours ago by sprt
Interesting, wonder if he's looked at the data from levels.fyi. Although it's surely not representative.
8 hours ago by borramakot
All the offer's I've gotten have been pretty solidly in the bell curve of what levels.fyi indicated.
10 hours ago by simonw
Love the section in this about using "boring technology" - and then writing about how you used it, to help counter the much more common narrative of using something exciting and new.
5 hours ago by m463
But "exciting and new" is very often just lipstick on a pig.
and anyway:
Daily Digest
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.