>>>
The Weather Service held a public forum Tuesday to discuss the proposal and answer questions. When asked about the investment in computing infrastructure that would be required for these limits to not be necessary, agency officials said a one-time cost of about $1.5 million could avert rate limits. The NOAA budget for fiscal 2020 was $5.4 billion.
Buchanan, however, stated the actual cost to address the issue would be higher because the $1.5 million “would comprise just one component of what has to be a multifaceted solution.”
The officials at the forum also said that senior management at the Weather Service was aware of the relatively small cost of addressing the issue but that the agency faced “competing priorities.”
<<<
So, the cost to fix would be peanuts, but it's "not a priority". Sounds like pretty typical corporate bureaucracy against improving infrastructure and efficiency until they're forced to.
Using their numbers, it'd still only something like .2% of their budget, if the cost was 10x the $1.5m figure.
“It is not clear why the NWS is considering these harmful bandwidth restrictions given the massive scalability of content delivery network (CDN) technology, cloud infrastructure and other technology solutions that are currently available,” AccuWeather’s Porter said. “It’s truly unfortunate that the NWS apparently does not recognize that this proposal is 100 percent contrary to its mission and its obligation to the American people.”
What is it with weather services and incompetent IT? Here in Australia, our federal "Bureau of Meteorology" (BoM) was hacked by a state-sponsored attacker, and used as a stepping stone to get into the networks of other government agencies that had private links to BoM.
The money shot is: "As part of this, BoM is planning to upgrade its web presence over the next two years as part of a $31 million digital channels platform deal with Accenture."
Two years later, they still don't have HTTPS on their main page. What kills me is that the main site URL is actually reachable over HTTPS.
You get an error message -- using HTTPS and a valid certificate -- saying: "The Bureau of Meteorology website does not currently support connections via HTTPS." and then you're redirected to HTTP!
Apparently $31 million and a year of effort is sufficient to buy a $50 SSL certificate, but doesn't stretch far enough to actually deploy it...
PS: In case you're wondering if this is some sort of infrastructure issue: Their site is served via Akamai CDN.
> So, the cost to fix would be peanuts, but it's "not a priority"
But there are probably thousands of things that cost only peanuts and aren't a priority. If you fix them all they add up to not peanuts and then you don't have any money left and you didn't even fix any priority tasks!
Serving the public should always be a priority for a government institution. In the case of the weather, disseminating the forecasts is just as critical as making the forecasts in the first place.
These forecasts directly affect everything from air travel to crop planting/harvesting to sea travel. They are critical for our economic (and citizen's) ongoing health.
And the internet, for better or worse, is the primary means of dissemination for the NOAA. Failing in this is failing at one of their greatest duties, IMO.
Instead, government institutions which are funded by government, now serve government as their main client. Public, which pays government, not institutions, is not a priority. Effectively government became an entity which is not serving the public but only itself.
The “problem” is if NOAA directly provides data to the general public then an entire ecosystem of companies that simply gather NOAA data and pass it along to the public suddenly lose money. Those companies then try very hard to limit distribution of NOAA data by NOAA while maximizing the government spending on accurate forecasts.
It’s got nothing to do with government supporting it’s self and 100% to do with legislated corruption that’s dependent on public officials playing their role.
> Didn't downvote myself, but if I had to lay money on it, I'd guess that you got downvoted for not providing links to the raw NOAA data? Sources FTW.
Yeah probably true, I didn't post a source because I thought it was common knowledge which in retrospect I'm not sure why I thought that.
And I'm glad you enjoyed the links, when I first started looking into the raw NOAA data and PRISM[1] data it took a bit to figure out what to do with it. But I've found the GDAL library helpful for parsing various weather data formats, and there are a lot of them[2].
PRISM is not NOAA raw data. It's a product produced by interpolating data from validated weather stations, taking into account elevation among other things. For example, doing straight interpolation between weather stations without taking into account elevation can lead to some very wrong results. In fact, PRISM stands for "Parameter-elevation Regressions on Independent Slopes Model"[0]. Also, the high-resolution PRISM data is not free, as it's a data product of OSU/NACSE. It costs real $$$.
Aside from point, weather station observations, most weather/climate data is gridded and you are right, GDAL provides lots of tools for processing and transforming gridded (raster) data products. DevelopmentSeed (no affiliation) creates some neat tools and I'd recommend following OSGEO[2] and the FOSS4G conference.
In a nutshell, we instinctively assume others know what we know. This was probably true when we lived in small tribes on the Serengeti, but not so much in the global information age.
It takes practice, but you can grow the habit of always providing a little extra context without sounding patronizing, as well, at least in-person, a better sense for when people are following the conversation.
What helped me grow this skill was lots of pair-programming, because that daily pair rotation forces you to be really good at context-sharing on the quick.
It’s reasonable to separate publishing data and providing data to the public. Their website only provides a subset of their API data so that missing data is provided to 3rd parties and curious computer savvy users, but not the public.
> Their website only publishes a subset of their API data so that missing data is provided to 3rd parked and curious users but not the public.
The majority of the data available to download is superfluous to most people hence why only a subset is shown on any weather app or website including forecast.weather.gov or Accuweather or whatever.
> It’s reasonable to separate providing data to providing data to the public
The NWS actually does provide visualizations for most of their data, and at this point I feel you don't know much about weather data/forecasting and are just looking to argue online. I provided a bunch of links below that allow curious people to look through American model data without having to use a company or having to program their own software. This list doesn't even include free to use apps/sites made by companies. American weather data is easy to access and view for free, Where as European model data is harder to access because you have to pay for it.
It’s actually the reverse. I was annoyed I couldn’t validate that I was correctly interpreting some of their API data on their website.
My original interest was displaying each days forecast vs what it had been the day before, but it turned out less interesting than I had expected. However, it’s easy to get sucked into this stuff.
Given that General Aviation uses NOAA weather data, that alone makes the NOAA data more useful than almost any other source. They also provide data to our fishing and ocean transport ships.
And, it's been available to the public via the internet for at least two decades I know of at nws.noaa.gov (which now redirects to weather.gov).
NOAA's data is a stupidly valuable resource, even discounting the resellers.
You can just go to national weather service site which uses NOAA data and it's quite human friendly[1]. Also I don't understand how you can call the raw data not very useful, when it's literally the backbone of weather forecasting in this country. Every weather forecasting app uses the raw NOAA data, its there for you to use as well[2]. The hardest part of using the raw data is the learning curve related to meteorological programing, not getting access to the data.
By use I am referring to people directly using NOAA or other government apps/websites to directly access it without intermediates. Other companies repacking that same data bring very little to the table.
It’s the same issue where the IRS could in theory cheaply provide a user friendly tax prep software for ~1$ per taxpayer. However, tax prep software manufacturers have a massive incentive to avoid that. Further, they are going to lobby heavily for the issue which impacts themselves where most people don’t care about losing out in ~50$ per year.
> By use I am referring to people directly using NOAA or other government apps/websites to directly access it without intermediates. Other companies repacking that same data bring very little to the table.
Huh? Did you even look at the links I provided above? They are literally what your talking about. One is a free government run website that any person can use to get a weather forecast, the other is a free government run website where any person can download raw weather data and use any way they like. So no intermediates or companies required.
No. The second link isn’t something 99.9% of the general public will directly access. So in terms of directly communicating with the public without intermediaries it’s again useless to 99.9% of the population. The first link is useable but hardly designed to maximize views by the general public.
This isn’t some major expense, even 3% of their budget spent on communicating with the general public would easily cover several clean apps etc. Instead, it’s clear their largely a giant subsidy for private profit. Other government agencies charge for this kind of large scale data access by private companies to lower costs to taxpayers by removing subsidies.
> The first link is useable but hardly designed to maximize views by the general public.
So you admit that forecast.weather.gov is usable to the general public, but because it "doesn't maximize views" its some how invalidated? You said above that the public shouldn't need to use an intermediary to get a weather forecast and that's exactly what forecast.weather.gov is. And not only that but its faster and better than the for profit apps such as Accuweather, Weather.com, Weather Underground, etc.
> This isn’t some major expense, even 3% of their budget spent on communicating with the general public would easily cover several clean apps etc. Instead, it’s clear their largely a giant subsidy for private profit. Other government agencies charge for this kind of large scale data access by private companies to lower costs to taxpayers by removing subsidies.
3% of a multi billion dollar budget is a lot of money also why would they waste money on a "several clean apps" as you put it when they already have a working weather forecasting app and that money could be used to address issues like the bandwidth one described in the article above, or updating weather models? It sounds like you just want a fancy new app even though they already have a working one.
First I suggested these companies should be paying for API access. Alternatively, one app per device is multiple apps. It’s not about making weather.gov outcompete private companies, it’s about getting value for taxpayer money. People shouldn’t be watching adds or paying for apps that simply show them data they already paid for.
Hell their API’s are providing a longer forecast than the actual website. That’s silly.
> Hell their API’s are providing a longer forecast than the actual website. That’s silly.
forecast.weather.gov doesn't show longer than 7 day forecasts because forecasts beyond 7 days are not very accurate[1]. So why show something that has a 50% chance of being wrong?
We're not discussing serving the public, we're discussing serving for-profit companies who make money off of the NWS's data. Those are the only people this policy change would affect, not you or me going to the NWS website.
This. Accuweather and weather.com have literally leeched all the data they can get from NOAA and then they resell it at a tidy profit. This change does not affect the general public but does affect the big players.
I was on the call. my work uses these data and we needed to know how things were changing. Not only external users but also some internal apps n such. One guy from a GIS section talked up and was like, hey is this going to affect the maps on water.weather.gov and other sites? The head dude was like, yup it is. So even internal agencies are getting affected by it.
Overall tho yes. Some large players take the data here , process it, and resell at larger value. Some might call it a value add. Accuweather is shit tho so dont ever go to them for anything
I think you need to consider whether it is the public who should be served in the free market or well connected lobbyists who have been highly paid to corrupt the executive branch. The NWS will not serve the public with what they trivially could so that a private corporation can make (more) money filling that niche and continue supporting politicians and their friend's lifestyle. That is the crony-capitalist way and corporate first amendment rights to spend money any way they want shall not be abridged (by this Supreme Court).
Because if there is one thing we know, government agencies are never captured and always have priorities well-aligned with the constituencies they serve.
I do understand that the peanut gallery normally has very skewed, ignorant views of what complex agencies do. I also understand if you don't question, audit, and occasionally yell at them, they go bad and (at best) stop serving the people they're supposed to.
> If you fix them all they add up to not peanuts and then you don't have any money left
The initial criticism is worthwhile in its original context (0.02% of the agency's budget). By scaling it up, you took it out of that context so that it runs up against worries about the total budget, implying a shortfall even though the argument is about the same 0.02%. The obvious answer to your comment is "well then it's easy to cut a little from $ELSEWHERE", which will inevitably devolve into overt zero-sum partisan bickering.
No, I was just looking a step ahead. The only thing your original comment added was an appeal to destructive penny pinching that often accompanies popular fiscal conservatism.
For context I'm personally against wasting money. But kneecapping an organization by tightening its purse strings such that it can't fulfill its function is itself a larger waste of money.
> Or why you're moving the goalposts to "reduce their budget".
Nobody mentioned this but you! It's entirely something you brought up and you're the only person with any opinion on it! I haven't expressed a single opinion on reducing their budget or what their budget should be! I haven't moved the goalposts to it because I didn't say anything about it!
> arguing about deep cuts ends up inherently partisan
You're the only one arguing about deep cuts, or any cuts at all! I haven't given any opinions on any cuts! I've said absolutely nothing about it! There's no competing opinions on this with which to argue!
You're not arguing with me you're arguing with someone completely imagined in your head! If you think I've got some opinion try to see where I actually said that before replying again.
You keep saying that I am imagining what I believe you said. From my perspective, this is gaslighting.
I will paraphrase what I think you said (and the straightforward context/implication). Please tell me where my understanding goes awry.
> But there are probably thousands of things that cost only peanuts and aren't a priority.
There are many other items that similarly need fixing (across all federal agencies). Each may not cost a lot individually.
> If you fix them all they add up to not peanuts
But if they were all fixed (applying a universal standard), the total cost would end up being a lot (spending a lot is bad, regardless of how much has been fixed)
> *and then you don't have any money left and you didn't even fix any priority tasks!
The total budget should not be increased, so a policy of fixing small things would take away significant spending from other things that are more important.
What’s happened here is you’ve seen an argument for ‘no budget currently available for’ which is just factual, and then you’ve assumed a lot more.
The bits you imagined out of nowhere is the opinion ‘and should not be increased’, and then you’ve even further imagined an argument for ‘cut’. I never argued those things. Go back and try to find where I argued for either of those.
Can you link to them?
I don’t think you’ll be able to. You’ll find you mentioned it first in both cases and even then I haven't given any opinion on them!
Why do you think everyone is flagging and downvoting you?
Perhaps where we actually start to diverge is that I view stories like this as appeals for increased funding from Congress. If NWS were going to rearrange the budget internally, they would have already done so. Did you not mean your comment in the wider context of managing the entire federal budget?
You haven't directly argued for cuts, but I still maintain that reflects the gist of your comment. "You don't have any money left" is an assertion of limiting the budget. The immediate implication is that increasing funding to something means having to cut something else. The cliched response pertaining to such tradeoffs in the federal budget is "well then, buy one less F-35".
The directly opposing argument is that would be worthwhile to increase the federal budget and taxes by 0.2% to fix similar problems across the board. I'm certainly not making that argument to increase taxes though - but we should be able to hold a critique of the government's foolish penny wise pound behavior in our heads without collapsing it to one of the two.
My comments were downvoted because HN abhors sarcasm, the exchange has generated much more heat than light, and that got attributed to me because I went meta. Next time I'll just copypasta the comment about the F-35.
Accuweather and the rest have been lobbying to hobble NOAA and NWS for decades
So of course it’s corporate bureaucracy. Like with the Post Office, corporate stooges are given control and run it like shit to have something to complain about.
I have written several command-line weather programs for different services -- many of which I had to abandon. For example, I wrote a popular one for Weather Underground that I had to retire because WU stopped allowing free access to their API.
My latest effort is one that just deals with NWS/NOAA directly, and reading this article has me wondering if part of their bandwidth trouble is a product of the way their API works.
The API is a huge set of linked JSON documents. So it very often happens that a request for data is actually half-a-dozen requests through embedded URLs (all of which are made by the client one after the other). It's all very logical in some ways, but at the same time a bit byzantine. I often wonder which entity is served by the way it's set up. From my perspective, it's easily the hardest API to work with. Weather Underground was wonderful, and so was Dark Sky (which I played with for awhile).
I've had people over the years ask if one or another of my tools could be used for large scale or very rapid weather data collection, but all of the services I have worked with throttle connections exactly as NOAA is proposing. I suppose I'm at least relieved to hear that they're not restricting access by independent developers!
You seem to know a lot about weather, so apologies if this comes off as a stupid question.
What value add do companies such as AccuWeather and others provide over simply going to weather.gov? Is their UI simply better, or do they run their own weather models? Thanks
Companies very rarely run their own weather models. This is because the models require a ton of computing power. Ex: the USAF current weather model is run on "Thor"[0][1]. It's a .9 PFlop system. And it uses approximately all of it, on a consistent basis. If it didn't, they'd increase the model resolution to match, or run the model more often to incorporate observed data faster.
Most of the companies like AccuWeather et al provide, is better UI, and tailored weather. Not just there's going to be a severe thunderstorm, but "these truck routes are going to be affected by weather, so expect them to be 40% slower than normal" or similar.
I hate wunderground - it is a giant, bloated mess of a website with a UI that jumps all over the place ...
However, I can't find anyone else to give me a 10 day forecast. It seems like wunderground is the only place that collates it like that, which I like very much.
Am I wrong ? Is there some other source for a proper, detailed, 10 day forecast ?
I just use https://www.weather.gov/ because its not bloated. Unfortunately it doesn't have 10 day forecasts and I believe the reason they don't show 10 day forecasts is because they are not very accurate. That far out your better off looking at forecast ensembles and looking at what the Euro, Canadian, as well as the GFS model is forecasting. But no common, free weather websites I'm aware of exposes that info. If you live in the west you can use this site to a degree[1], although its more targeted to snow/ski forecasting.
Is it really difficult to find 10 day forecast? Windy is a particularly good app for these kinds of things.
I also run a small REST API service (oikolab.com) that provides location-based forecast data out to 10 days using NCEP service. I started out using DarkSky and Wunderground a couple of years ago but couldn't really get the type & quality of data I needed so I ended up creating one from the original raw data.
It is a 're-packager' in a sense but what we bring to the table is the ability to query the data in a time-series manner efficiently for specific set of end-users. The way NCEP's GRIB data is packaged makes it very slow and painful to do this if you wanted weather parameters to feed into other analytical models (e.g. building energy simulation, time-series forecast etc).
Im in the same boat, their site performance has seem to just been getting worse over time and it is barely even functional on my phone, however I don't know of anyone else that condenses almost the entirety of a highly detailed 7+ day weather report into such a small and useful little graph with every detail you would want. What would take multiple pages worth of reporting is covered in a small graph the size of my hand.
It was invaluable when I did outdoor work, everyone else was making guesses and extrapolations based on the shitty reports from different local news stations and radar and satellite maps, meanwhile I could tell exactly when a big storm would role through and if rain would be spotty or temperature swings with just a glance for multiple days ahead.
Look for an app which lets you download GRIB files. PocketGrib and LuckGrib are a couple to try. You can download GFS data (produced by NOAA) for the area/timescale/data you are interested in, drop a cursor anywhere on the map and view a meteogram.
This is the same data used by "value add" weather forecasting sites and services to produce their forecasts.
Accuweather will give you a 3 month forecast! I had to blink a few times to make sure I got that right. Their existing long term forecast goes to March 10, 2021.
It's worth being aware of its shortcomings, as even its 25 day forecast has been shown to be no more accurate than climatological averages.[1] They also have done some shifty stuff in the past regarding data privacy, among some other weird things.[2]
I have a degree in meteorology. I can forecast weather for the 4th of July over next 3 years (2021, 2022, and 2023). here in Chicago, Illinois.
will be upper 80s, low 90s. Chance of thunderstorms in afternoon.
Congrats, i have now predicted a typical 4th of July and will be correct more than normal. Damn i should have built a web company and sold my ideas for a bajillion dollars
But I still appreciate that the Dark Sky feature isn't artificially limited to a particular date. They have a nice error bar on the temperature that gives you an indication of how much it's unknown.
There is a text box in the top left which says "Local forecast by "City, St" or Zip Code". Enter your city, state, or zip code in that box and select the location. Then it opens the forecast page. I have included a link to San Francisco below. You get a full 10 day forecast. I'm not sure what else you want.
You must be counting the day and night forecasts as separate days. There are no 10-day forecasts on the linked page. The furthest-out forecasts are 7 days into the future.
A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations.https://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypert...
The article mentions rate limiting by connections:
> limit users to 60 connections per minute
You can make multiple requests over a single connection.
> allow servers to instruct clients on how to construct appropriate URIs
Right. That would be great, and actually, that's what the commercial services do.
With NWS it goes something like this: Say the user has a lat/long. You fire off an http request for a document that tells you what "quadrant" you are in and the URL for it. You then fire off a request for the quadrant document, that lists the products that are available (and their associated URLs). You then fire another request for the actual product for that quadrant . . .
This is not a real example, but this is basically how it works. Even when you can construct a single URL with the info you need, you still need to make multiple requests to have enough information to build the URL (if that makes sense).
> You can make multiple requests over a single connection.
You mean with some kind of http keep-alive thing? I'm not sure I can do that in these circumstance (with cURL).
Jonathan Porter, a vice president and general manager at the private forecasting firm AccuWeather, warns that the agency’s proposed solution would harm the timeliness and accuracy of forecasts and severe weather warnings. He said the collection, processing and distribution of weather information are the agency’s “most important services.”
"Richard Hirn, an employee who represents the National Weather Service’s Employee Organization, told Bloomberg, “We fear that he [Barry Lee Myers] wants to turn the weather service into a taxpayer-funded subsidiary of AccuWeather.”"
AccuWeather is in the business of spreading FUD about the NWS. They are directly responsible for the current level of underfunding for their digital products.
The headlines are misleading. The people leading the call actually said the problem was 'volume' of data. I heard them talk, i'm not sure they really know what is going on or understand the issue.
Also the problem isnt just weather.gov and weather API. Sites like accuweather are likely grabbing raw files from the FTP servers. Everytime a new forecast comes out (for say the GFS), all these people hit the server at same time to download gobs and gobs of data. Yall would be surprised at how much data in is weather prediction outputs!
The CEO of AccuWeather Barry Myers who Trump just appointed to head NWS/NOAA is essentially trying to privatize it. NOAA's API is horrible, slow, and you often have to resort to scraping the HTML because it leaves out data. Instead of rate limiting essential 3rd party services they need to rewrite the API and heavily rely on caching and a CDN. Sadly it looks like private money lobbying is ruining an amazing public good.
The NWS radar web interface is one those great examples of a government service done right. They've been working on an upgraded version, the public deployment was just pushed back a week. The new stuff appears much more bandwidth intensive, especially since "shared cache" is dead.
https://preview-radar.weather.gov/ still seems to be getting assets off arcgis.com ; surely theres room for a public map server that can stand this traffic?
I haven't looked at the code; I suspect its amenable to hacking up one's own views.
I think the new site is horrible and bloated and slow. I've been using their multi sensor page for a couple of years now and think it is much better: https://mrms.nssl.noaa.gov/qvs/product_viewer/
Completely agree. They are even using tiles for the radar image which makes it look awful and blocky because they don't all load properly.. unlike the local radar of current version which is the entire image of that radar site.
AGOL (ArcGIS Online) is only providing the basemap and layer overlays (state boundaries etc), those probably come from Esri's "public" data portal and I doubt are part of the bandwidth issue.
Yup. I can't even access the new version. I've attempted to email them asking if they'd keep the perfect content-centric HTML-as-a-document site up but never received any responses.
The animated displays are miserably slow to load and don't perform very well. I'm in the habit of using the radar lite version of the old system, which has the whole gif loaded fast enough for the first loop to not be delayed.
On the new site, the base map tiles start to filter in after about the same amount of time, along with a few pieces of the first radar frame.
The political motivation is to portray the government as incapable of doing anything other than defense, so that the functions can be outsourced to the private sector.
Expect a solution to appear shortly after the regime changes.
The people who want to wreck the NWS don't want it outsourced to them. They still want the government to pay for producing all of the weather data and forecasts, then they want the government to designate them the exclusive distributor of that information so they can charge whatever they want.
You still have to pay for those CDNs. If you think this is a red/blue issue you haven’t been following along, though.
Obama allocated funds specifically for satellites, but reduced the operating budget [1]. Trump actually increased funding for several areas as part of the stopgap funding in 2018, particularly for satellites [2].
There are other examples over the decades of course, but something that seems to be a trend is the specificity. Satellites are flashy and exciting and get funding from politicians. Servers? Meh... why don’t they just get a couple extra machines? My laptop is fast as hell and only cost like $600. Bandwidth? Why don’t they just get whatever internet I have at home? That costs like $100/m and the Por— I mean the CNBC videos start immediately.
This why I'm so excited about Cloud in the public sector. Its so easy to defer upgrades on a server fleet, even though it might cost someone more money later on.
The Public Cloud provides a monthly expense that just becomes part of the operation budget.
It's exciting to think about the innovation to government services as an end-result.
> The Public Cloud provides a monthly expense that just becomes part of the operation budget.
Ops budgets aren't infinite. In this example, if the cloud egress costs kept rising due to demand then the weather service would still have to find ways to restrict usage.
Remember with NOAA and NWS you always have GOP pressure originating from Accuweather, which really wants to be a the gatekeeper for this data.
Over the years they have attacked the NWS web properties in particular (Rick Santorum was their attack dog a few years back) as their business of selling graphics to TV stations has declined along with traditional TV.
Errm, AccuWeather is one of the companies complaining about this proposal. From the article: "Jonathan Porter, a vice president and general manager at the private forecasting firm AccuWeather, warns that the agency’s proposed solution would harm the timeliness and accuracy of forecasts and severe weather warnings. He said the collection, processing and distribution of weather information are the agency’s “most important services.”"
Apparently not getting all that NWS weather data for free and then being abl to use it in their commercial weather forecasts would put a bit of a dent in their business model. Actually, I think commercial organisations like AccuWeather probably be the main ones affected; ordinary end users aren't going to be requesting large quantities of data.
So NOAA and the NWS are playing 3D Chess against AccuWeather whose been loddying against them for years? Now, unless AccuWeather wants to start paying for the data they'll need to start lobbying for them... I love it.
So basically, AccuWeather wants to have their cake and eat it too.
What's the nature of their lobbying?
Are they basically trying to defund any kind of front-end, consumer facing services from the NWS/NOAA, while keeping the back-end data sources available for them to feed their own consumer facing services?
That is the specific scenario. They've lobbied for the NWS to not run any public facing services, but still collect all of the data, run the models, and put the GRIB files up, but then stop there so that Accuweather can monetize it.
> The NWS was giving away forecasts on its website, radio stations, and elsewhere, when businesses such as AccuWeather charged its clients for theirs—never mind that AccuWeather relied on the service’s free data to formulate its own predictions. Santorum agreed that commercial weather companies deserved protection.
Well, not if you don't understand how hard accurate weather forecsts are to get, or why a private entity couldn't do better. After all, that's why the news shows have weathermen, right?!
And if the NOAA goes away, the government becomes smaller, and their taxes go down, right?
:/
EDIT: Dear downvoters - while this is not my opinion, this is how people think, what people believe. Dismiss it at your own risk.
NWS is an amazing resource for pre-trip planning for a variety of different people. I frequently consult it for weather forecasts prior to hiking. I'm not sure a company like Accuweather could provide me with access to such great forecasts or sensor data as NWS does given how reliant they are on NWS data themselves.
The other problem here is that Accuweather charges huge fees for access to their forecasts and I'm not sure that I, as a private citizen, could afford that. So my choices go from pay a little bit of tax to get access to high-quality, reliable weather forecasts and warnings, to being totally unable to get access to this information without paying exorbitant fee. Or alternatively just not knowing these things and having to pay higher insurance costs for my ignorance.
This is a really good example of how making government smaller makes a lot of peoples' lives much crappier.
Yes, you're right, I over-simplified. Many of them are (or have) meteorologists. To give the data itself a pretty face (interpretation). But that's not the primary reason that news stations hire "weather people". If it was, they would just contract out to some other company.
> I don't care who you voted for, this should make every American really, really mad.
There exists a certain group of people that think private sector and The Market™ should handle many/most/all things. Do you want to guess which political party those people tend to associate with?
Also, should we be more or less mad about this than someone trying to overturn fair elections like is happening now? Do you want to guess which political party is trying to do that? Do you want to guess if it's the same political party which believes in The Market™?
So you may not think it matters who people voted for, and that this should be a universal feeling… but that's not what reality is. Do you know what other things should have universal consensus?
First, there's no way to compare the election stuff with this weather thing. Sometimes different things happen at the same time for different reasons.
>There exists a certain group of people that think private sector and The Market™ should handle many/most/all things
I think the Accuweather problem is worse, much worse. What has happened is that NOAA & NWS had been happily supplying data to..whoever..when some people decided to create a commercial service with the same data. Then they hired disgraced Jesus-freak Santorum to play lobbyist and try to kill the NOAA/NWS public services. Note that Accuweather is not going to fly their own satelites, they just want you to be locked out.[1]
That's right: if a private company decides to commercialize something the government is already doing, using the same government services that that government function is using, then the government should stop doing the thing and let the private company have the market.[2][3] This is a continuation of the Republican principle of privatizing profits and socializing risk, as well as colonizing publicly-funded resources.
I'm not arguing with this view, but I want to emphasize, that as someone who once worked in the Federal sector in numerous different agencies, that they waste tremendous amounts of money on things that have nothing to do with their core mission. The Pentagon is obviously horrendous, but other agencies that I expected to be better stewards of their dollars (EPA, DoE, IRS, and yes, NOAA) were horrifically wasteful, on dozens of fronts.
The hiring is always a problem, even when there isn't a hiring freeze, but they don't fire enough lazy people, which demotivates hard workers who end up bailing for the private sector.
As taxpayers, we should be furious at the GOP for the Accuweather bullshit, but we also need to be furious at any government agency which tolerates the level of fraud, waste, and abuse that most of these agencies perpertrate.
As a person who happens to want universal healthcare like the NHS in Britain, it's infuriating to me that these agencies tolerate such constant mediocrity in their ranks (mixed in with the talented of course!!!) and are so bad with their spending efficiency.
Bezos had something to say about the Seattle city government last year:
"They don't have a revenue problem, they have a spending efficiency problem."
It seems like we always get political about this, with one side wanting to "starve the beast" (as if government is this evil beast), and the other side just always letting them get away with the excuse that they need more revenue.
Where are the people who BELIEVE in government that will demand that it be efficient?
Edit:
I love how people think that because corporations are inefficient, they assume that all organizations are equally ad inefficient as the US Federal Govt. The rate at which people are involuntarily terminated in the Fed Gov is far, far too low, because incompetence is routinely tolerated where it wouldn't otherwise be anywhere else. Why? Not because a person's supervisor in the Fed doesn't recognize the incompetence, but because the firing process is so difficult that they don't want to or can't expend the energy to do so. If you want government to have a bigger role in people's lives, but are so delusional as to not recognize how bad it is, then you aren't furthering your cause.
The FDA still hasn't approved the Pfizer vaccine, while the NHS already did. Pfizer is an American company. That's a national embarrassment.
None of those inefficiencies are unique to government agencies. They all happen in the private sector two but you don't hear about them because corporations are not subject to an open records act. And that "oh but the free market will ensure inefficient companies fail" response some are about the write is total bs. Companies fail for all kinds of reasons, inefficiency is just one of several things and not always deadly to a corporation.
I’ve worked in both. Companies are dumb in different ways than government.
Usually companies have better leadership and are more merciless with people. Unless they are unionized, problems get pushed out (unless they don’t). Big companies waste lots of money on other things.
Government agencies have by design a split between political/professional management. Sometimes that results in strange stuff. Also the programs in government are very meaningful — legislation limits how money is spent, one unit or function may be drowning in funds, another may be dumpster diving.
Companies waste money on things like exec compensation, marketing, etc.
I have. State government too. Also a couple BigCo orgs. I can confirm that all are absurdly wasteful. The incentives, culture, and flavor are different but the results are the same.
I've been a on several "tiger teams" (super cheesy, i know!) that were parachuted in from a consultancy to bail out Federal Gov projects that had gone bad. I've been tasked to state governments as well, and was always blown away by the huge variation between different states. Virginia, for example, at least around 2010, had a very well-run government. Illinois is a steaming pile of garbage as far as being a black hole for money. New Jersey is even worse. They recently, in the beginning of the COVID crisis, blamed their COBOL system for the unemployment check backups. The expert team that was brought in quickly determined that the system wasn't the issue, and the bottleneck was bad management.
I think that, like governments, companies are not all equal in their efficiency. I work with a several people who used to work on the factory robots at Ford, and the level of efficiency at Tesla dramatically exceeds that at Ford. As organizations age, they get more and more corrupted, unless reforms are periodically pursued. Think about what IBM used to be, vs what it is today.
I think the problem with the Federal Government agencies is that they started out very efficient, and filled with bad asses who were hungry. They are never actually reformed though. They just go through periods of politicians who starve them of money then to other politicians that give them plenty, and there's no consistency party wise of course. GOP throws money at their pet agencies, Dems at theirs.
But starving them of money and then throwing it at them again isn't any kind of reform. The result is they have remained stagnant, and filled with people who substitute meaningful work for work that generates great optics.
In late 2017, I was chosen to be on a team to go engage in a Hackathon for the Department of Health and Human Services that was focused on developing solutions to help combat the opioid crisis in the US. My company actually pushed to organize it, and therefore my team wasn't allowed to be considered for a prize, but we did participate.
The rules were that the teams would all get in the building, sign NDAs, and then be given data that wasn't available to the public, and have 24 hours to build a solution. We walk in, and they had changed the name to a "Codeathon", because high ups in HHS didn't like having the word "Hack" in the title. They had a massive room filled with suits, watching presentations about the opioid crisis and what HHS was doing to solve it, and pushed the actual competitors into side rooms that were incredibly uncomfortable, with card tables and folding metal chairs. We then ended up having to wait because they had delays in getting the USB drives with data out to the contestants. The person whose job it was to make the drives forgot to do it the week before. We waited for 5 hours for the drives to be distributed.
Eventually, we got the data, and started the work. Built some very cool solutions and presented them. The suits who had been sitting in the main room watching the presentations by HHS didn't show up to watch the presentations, because they were at 3PM on a Thursday, and we were told that the traffic in DC is bad, so most of them had bailed out back to Virginia and Maryland by then.
The hackathon (or codeathon) had no food provided. We had to periodically leave the building, buy food, and re-enter through the security checkpoints.
The sad part is that unlike me, the rest of my team had never seen the Federal Gov in action. They were all shocked at how incompetent everything seemed to be, and how many people in fancy suits seemed to do nothing. Hundreds of them, watching presentations. They had a long discussion on the flight back about how dismayed they were with the whole experience.
1. It sounds like you've mostly worked with government agencies, suggesting a selection bias in comparing public v private inefficiency.
2. The 2017 event you say was marred by incompetence occurred during the Trump administration, which was well known for being incompetent not at just governing, but at basic functions running an Easter egg hunt. https://www.nytimes.com/2017/04/11/us/politics/white-house-e...
When a company wastes their shareholders’ money, the shareholders have an easy path to sell their shares and stop being harmed by the waste. When a government wastes taxpayer money, there is not the same timeline nor precision in ability to stop directly supporting that waste so you reasonably see taxpayers seeking other avenues to attempt to curtail the waste.
“The Weather Service’s proposed remedy is to limit users to 60 connections per minute on a large number of its websites that provide weather observations, forecasts, warnings, computer model data, air quality information, aviation weather support and ocean conditions.”
Well that sounds incredibly reasonable in general, but it completely depends upon what type of interface they’re providing. Do you have to query all of those things separately for every locality you serve?
Honestly, this is why so many big companies still prefer things that HN would call “stupid” like an FTP drop of a file 5 times per day/hour/etc.
Every FTP implementation I’ve ever seen has included either checking the timestamp, timestamping the filename and filtering on that, or moving to an “archive” folder after processing. Most of the API implementations I see ignore the fact that HEAD requests exist, cache headers can and should be used, etc.
You have any experience with numerical weather prediction files? The ftp files you download are typically 'the whole globe' with a bunch of variable stacked like layers of images in a big image file. One of the problems is that a typical numerical weather prediction output has so much fucking data you dont need.
GFS for example. IT has maybe 200 or so timesteps in each forecast, runs 4x a day. There are a shit load of configs to run for the GFS. And each config + output has maybe 300/400 variables in each file? So you download 200 files, each with 400 images in it, each image is pretty dense. Each file is ~200 megs, X 200 files. You do the math. Get that 4x a day. And that is just one forecast model. We have lots more, and many more that have higher resolution and denser data.
The biggest problem i think is they have a goddamn free-ass oldschool FTP server. They are allowing dimwit grad school scrip kiddies to code up whatever shit they want. Infinite loops to try and download gobs of data you dont understand? sure go ahead.
They dont even know how to trakc users. they want to keep an open server, and use IP Address as unique ID. I suggested to use fucking tokens, or username/pass combo at least haha
Heh. Well, perhaps, but for public data it doesn’t really matter in general, does it?
Just like that installer you just downloaded off the internet, did you check the md5/sha to make sure it wasn’t tampered with? Just because it was over HTTPS doesn’t really mean much.
I think there are two key solutions for a resolution here:
1) CDNs are a tried a true method for getting content closer to the end user, properly cached, and for less cost.
2) Expose more services that NOAA and the NWS delivers as APIs. If you are providing services for the public good where the consumers are just scraping Web pages to gather the necessary data and content, then you are doing it wrong. It sounds to me like they are serving more API type content over Web pages than they are Web content to Web consumers.
I don't want to second guess the NWS/NOAA technical architects, because I know they are probably doing the best they can. Someone simply needs to communicate these goals and solutions to the political appointees...
This has been done for decades - long before HTTP took over as the standard we had sites like ftp.cdrom.com and ftp.kernel.org. Usually this is done by using a DNS that either is localized to your or round robin DNS. There were many times when I remember logging into ftp.kernel.org and being surprised that I was connected to a server in Finland, Europe, or Japan. Wasn’t the smartest thing, but with more advanced DNS services, it certainly seems possible.
Yeah it’s not difficult to implement. The question is, are there offerings that are as easy to set up like Cloudflare?
If you implement it yourself you still have to set up geographically distributed servers and a syncing system, as opposed to registering an account and changing a pointer in DNS.
Likely this is on purpose. I remember some congressman (PA's Rick Santorum?) trying to forbid the US Weather Service from sharing weather reports/forecasts so a partner of his would have a better market to sell the same service into.
There's a lot about this in Michael Lewis' "The Fifth Risk". The most maddening part is that the company lobbying to weaken the National Weather Service can't replicate much of what the government is doing here and sells a lot of repackaged NWS data. I suppose it doesn't matter to them if dismantling the service leads to less weather data being available overall, as long as their piece of the market gets bigger.
Users are accessing data in high volumes, especially during unusual weather events, which brings down the services. Is there something I'm not understanding about the problem here with bandwidth? Does cloudflare or another CDN for this stuff really cost an amount hard to include in the budget for such an essential public service?
Or, shit, can cloudflare just go and donate this to the government? It would be pretty good publicity for them and probably not a huge cost on their scale.
What am I missing? This seems ridiculous that the USA cannot afford to provide this high-value public service. When did we become a two-bit country when I didn't notice?
This is a great time to check out Michael Lewis's The Fifth Risk (https://www.amazon.com/Fifth-Risk-Michael-Lewis/dp/132400264...), which does a deep dive into NOAA, the weather service, and the wretched attempts to strangle the usefulness of this public service while keeping it alive enough to act as a massive subsidy to accuweather and related bottom feeders. It should disgust both libertarians and proponents of active government, and really anyone paying attention. Accuweather wants NWS to do all the forecasting work but then funnel all the data through Accu and a few other private hands, who of course want to profit from this data but not pay for it.
I've had some experience dealing with the Weather Service when trying to develop a weather app using https://www.weather.gov/documentation/services-web-api#. It was a frustrating experience and my initial impulse was to vent about that. But I'm choosing not to do that mainly because it seemed at the time that every person that I dealt with at NWS was trying to do the best they could.
The situation that NWS is facing is basically that massive tax cuts coupled with growth in defense and social programs has left most other areas of the federal government hollowed out. The amount of money needed to fix the problem being discussed here is minimal but after decades of budget cuts across non-defense and non social-welfare programs there is simply little left to invest in improving services.
>>> The Weather Service held a public forum Tuesday to discuss the proposal and answer questions. When asked about the investment in computing infrastructure that would be required for these limits to not be necessary, agency officials said a one-time cost of about $1.5 million could avert rate limits. The NOAA budget for fiscal 2020 was $5.4 billion.
$5.4 billion is indeed a huge number but gives no information as to the availability of the $1.5 million. Without knowing what the cost is of what they must accomplish with the $5.4 billion no conclusions can be drawn.
I question the accuracy of this article. I'll have to poke around for notes from the I2 TechExtra a couple of months ago, but NOAA was there and talked about what they have in the works.
I am by no means an expert in this space, but this was my first thought as well.
A government agency with bipartisan support and positive public image seems like a good entity for Cloudflare, Google, Amazon, Microsoft or someone else with infrastructure and bandwidth to be a white knight for.
The government isn't allowed to accept gifts to avoid the appearance of bribes. They can use a free service or offering but it has to be free for everyone
> According to companies that draw large amounts of this data, the proposed limit will substantially harm the services they provide to customers. The possible negative effect on forecasts has also raised concerns among congressional lawmakers.
Well then maybe they should be paying for more efficient feeds instead of querying more than once per second for a sustained duration? Really? In what world did anyone think a good solution to ANY problem was "query a website over 60 times per minute continuously"???
Yes, the former CEO of AccuWeather was appointed but not confirmed to lead NOAA. He was not a scientist and had spent years attempting to undermine the agency from the outside.
He also, in keeping with tradition for Trump appointees, was involved with various scandals.
"key data" .... which would be a small subset of all data ... and to solve bandwidth issues they want to make it more scarce.
This is a stunt. They want money, and are using their most popular (key) data points as a threat.
Just like politicians. Money gets tight and they say they'll cut essential services like fire and police, but what they won't tell you is they really want their pet project funded, using deceit.
As far as I know, the weather.gov api is hosted out of a single facility (the NCEP NCO), and if that building has any sort of network issue, the api goes down. At the very least it seems like using some sort of CDN would provide a bit more resiliency, but maybe there are infrastructure constraints that make this difficult.
Does anyone know where one can get historic wind data (avg direction and speed per hour for example) for a well known location like LAX? I was looking a few months ago and couldn’t find anything. I thought surely NOAA or NWS would have this?
YEah i can tell you for 10 bucks. Try to google wind speed LAX for starters. Did you search at all? Airports have weather stations that record shit and you can get at those data.
As mentioned in the comment elsewhere in this thread, I run an API service (oikolab.com) that does this. You can get hourly data for any location from 1950 onward.
I've suggested multiple times to them and even offered to help them setup HTTP BitTorrent to reduce the loads, but there's just no actual interest in solving the problem...
NWS could push the data commercial users want into a requestor pays S3 bucket. Archived/historical data could be rate limited, and made available for free (and I'd probably upload it into the Internet Archive for good measure).
There is a solid case that citizens requesting weather data should have such requests fulfilled at no or minimal cost, similar to GPS services (a public good) or FOIA requests. The government does charge user fees for some commercial services (IRS tax transcripts for income verification, for example).
EDIT: Please take the time to submit a comment on this issue [1] (last page of the pdf, December 18th deadline).
> There is a solid case that citizens requesting weather data should have such requests fulfilled at no or minimal cost, similar to GPS services (a public good) or FOIA requests.
I totally agree that weather data should be provided for free as a public service. The NWS is one of the true gems of the US government.
That said, this view isn't universally shared. Many other countries' meteorology services charge for access to data - especially if you want access to the raw model output. Most famously, the US's GFS output is provided for free while the EU's ECMWF output requires a paid subscription.
If open access to weather data is something that's important to you, please make your voice heard!
The EU has the ability to copyright things in general, which is horribly wrong IMO.
This is one of the few areas I'll defend the US to the bitter end on. Content produced by a government can not be morally copyrighted. The funding comes from the people, and therefore the copyright belongs to them too. Another way of putting it is the public domain and government copyright are the same thing logically.
There is, but it has its own downsides. Also, charging for internet services would have the effect of cementing the monopolies: why pay $5 for signal and $5 for mastodon when I can pay $5 for Facebook and get both?
>>> The Weather Service held a public forum Tuesday to discuss the proposal and answer questions. When asked about the investment in computing infrastructure that would be required for these limits to not be necessary, agency officials said a one-time cost of about $1.5 million could avert rate limits. The NOAA budget for fiscal 2020 was $5.4 billion.
Buchanan, however, stated the actual cost to address the issue would be higher because the $1.5 million “would comprise just one component of what has to be a multifaceted solution.”
The officials at the forum also said that senior management at the Weather Service was aware of the relatively small cost of addressing the issue but that the agency faced “competing priorities.” <<<
So, the cost to fix would be peanuts, but it's "not a priority". Sounds like pretty typical corporate bureaucracy against improving infrastructure and efficiency until they're forced to.
Using their numbers, it'd still only something like .2% of their budget, if the cost was 10x the $1.5m figure.