Caveat: OGFMirror

[The below is cross-posted from my other blog.]

I’m super bad about posting to this blog. That’s partly because I feel a strong desire to report some actual, positive progress, which I haven’t felt enabled to do.

I have been very busy with HRATE technicalities. I am building – very, very slowly – a “mirror” for (OGF). I think if this is successful, then the owner of that site, who has expressed interest in “letting go” of having to continue to maintain it, will allow the mirror to take over for the site and a transition to a new hosting environment will be complete.

Someday, I intend to write up, in elaborate, technical detail, this process of setting up a mirror. But in broad outlines, here is what it involves (has involved, will involve).

  • Build a new Ubuntu 20.04 LTS server. This leads to lots of incompatibilities farther down the line, because the existing OGF server is an older version. Install the basics – apache, postgresql, etc.
  • Install an OSM rails port on the server.
  • Migrate the OGF data to this server. This was very, very hard – because the OGF data (in either .osm.pbf format, or in pg_dump format) proved to contain inconsistencies (data corruption). Some missing current nodes and ways had to be restored manually (text-editing .osm = .xml files). This ended up a 2-weeks-long process.
  • Set up incoming replication from the source apidb (OGF) to the new mirror (currently being called ogfdev).
  • Set up outgoing replication for the new ogfdev instance (to drive render, overpass, etc)
  • Set up a new primary render. This had some sub-parts.
    • coastlines. This proved very difficult, because as far as I can figure out, the osmcoastline tool used to create the coastline shapefiles is broken on Ubuntu 20.04. An older version must be used. My current workaround: I’m actually running coastlines on an older server. I import a coastline-containing pbf file to the older server, run the osmcoastline tool, and post the shapefiles for consumption on the render server.
    • I made a decision to run the renders on a different server than the apidb. I think this might involve a bit more expense, short term, but it makes the whole set of processes more scalable, long term. My experience with Arhet is that the render requires scaling sooner / more frequently than the apidb, as the user base grows. Installing the render software (mod_tile and “renderd”) proved difficult. It turns out that there are some lacunae and downright incorrect steps in the documented installation sequences on github.
    • Set up incoming replication from the ogfdev database to the render database.
    • There are substantial differences in recent versions of the openstreetmap-carto style – specifically, the special shapefiles are no longer stored in as datafiles in data folder in the render directory. Instead, the shapefiles are loaded to the database. Because non-standard shapefiles are used, this means rewriting the load procedures (python scripts) – the standard approach is to just grab the files for “Earth” (because who would run osm for some other planet?!). So that file-grabbing is hard-coded in the procedure.
  • Set up a new topo render. The topo render was shut down on OGF, so this will be the only working version. Unfortunately, I ran into a similar problem with some of the topo pre-processing as I ran into with osmcoastline, above. I suspect for the same reason – something in one of the dependencies they both have. So the topo pre-processing (turning the .hgt files into a contour database) is also being run on a separate, Ubuntu 18.04 server (just like the coastlines).
  • Set up appropriate changes and customizations for the front-facing rails port (osm website). This involves importing user data (done) but also user diaries (not done). These require ad hoc SQL coding that give me flashbacks to my job as DBA in the 2000’s. Another unfinished piece – internationalization. The current ogfmirror website looks okay, but only in English. Switch to another language, and it all reverts to OSM boilerplate. Why is internationalization done so badly on production software of this kind? I see no easy solution except manually editing each language’s .yml file in turn (OSM has a 100+ languages). Or building my own damn application to achieve that result.
  • Set up overpass and overpass-turbo. Overpass installs relatively painlessly, but I’m having trouble getting incoming replication to work correctly. overpass-turbo was quite difficult – the current version on github is flat-out broken, and so an older version (commit) must be compiled and installed. Further, the compilation and configuration process overwrites some of the parameters files, so the parameters files have to be modified after running the first steps of configuration, but before the last part. This is the step I am on right now.
  • Set up nominatim? – nice to have, but not urgent. Anyway nominatim doesn’t work on the existing OGF website
  • Implement some of the custom tools that are available on the OGF website: the “scale helper,” the “coastline helper,”…
  • What else? This is a work in progress…

So I’ve been busy. Here is a link to the site. Bear in mind, if you are reading this in the future, the link may not show you what I’m currently writing about, but rather some future iteration of it.

I’m still working on some of those last steps. Open to hearing what else needs to be done.

What I’m listening to right now.

K-os, “Hallelujah.”

Caveat: Fishing Report #(n+25)

I’m a bit uncertain as to how to proceed, in the event Arthur goes out fishing but I don’t. This is my blog, not Arthur’s. So my gut intuition is not to include reports of his excursions. But I also wanted the fishing reports to be a log of our “take” and where we got results. So for that reason, I want to record it.

For the record, Arthur went out fishing today, with his brother Alan and with two guests – Joe (who has joined us before) and Joe’s friend Jim from Idaho. With the four of them, I felt that the boat would have been too crowded with me along, too, so I figured they would work it out, among themselves. I trusted Alan, Joe and his friend to competently take on my role as “safety officer.”

They caught three coho off San Ignacio Island. “All catching was done in the fog,” Alan summarized. They also caught a few small black bass and rockfish, thrown back.

Year-to-date totals:

  • Coho: 14
  • Kings: 0
  • Halibut: 0
  • Other: 1
  • Too-small fish sent home to mama: 17
  • Downrigger weights left on the bottom of the sea: 1


Caveat: Tree #923

This tree is from my past. It was witness to a rather ambitious hiking excursion I took with my brother in September, 2013, in southwestern South Korea – right during the time I was undergoing my 3-times-a-week radiation therapy for my cancer. This is on the mountain just west of Hongnong, between the town and the nuclear power plant on the coast. Hongnong is where I lived in 2010-2011. I remember being utterly exhausted from this trip.

picture[daily log: walking, 2.5km; retailing, 6hr]

Caveat: Tree #922

This tree was there when the sun came out. In the lower right of the photo, in the shadows, you can see the stairway to the treehouse.

picture[daily log: walking, 3.5km; retailing, 6hr]

Caveat: Digital Anti-vaxxer

I might have just converted to a role that might be called a “digital anti-vaxxer.”

My computer crashed this morning. It was far from catastrophic – I have good back-up habits. I lost a few dozen recent pictures, and some text files I can’t even remember what was in them. I may have lost some other stuff. Further, I have a perfectly good “extra” machine, which I am now using. It’s not as comfortable in its configuration, and will take some time to get used to, but it serves my basic needs fine.

And I knew that the computer in question was sickly – it was my HP “Lemon” laptop I bought in 2018, which has always had a bad battery and long had other issues as well. I have been using it as a desktop computer (because bad battery). I had changed the windows over to Linux. It wasn’t a terrible machine.

I believe the mistake I made, yesterday, was to let alarmism seen on the internet induce me to finally look into installing some anti-virus software on my Linux machine. I did a little bit of looking around and elected something called Clamav.

As background: I have never run anti-virus software on my Linux machines. And frankly, I’ve never had problems with viruses or malware on my Linux machines. Even on my Windows computers, when I’ve had and used them, I have never installed any kind of paid anti-virus, though for Windows machines I’ve occasionally run “system scan” or system monitors of various kinds.

I had always felt that with respect to anti-virus software, the cure was worse than the disease. But with respect to the principle, I could see where people who had less comfort and familiarity with the inner-workings of computers might have a reasonable use for anti-virus software. Or, barring that, they can get Apple products, which has the anti-virus buried inside it at such a level that it’s invisible and relatively non-disruptive to the user.

Anyway, back to my narrative: I installed the Linux anti-virus software on my computer yesterday. And, having never had a major problem with my Lemon’s software (only ever with hardware, before), this morning, I found the machine was “bricked” – this is a term used to describe computers or smartphones that simply cease, utterly, to work. Black screen, no boot, that kind of thing.

The only thing that I did different was install that antivirus software. So my conclusion: the cure was indeed much, much worse than the disease. And I am an ever more committed digital anti-vaxxer than ever before.

Which feels odd, since I’m not an anti-vaxxer with respect to human vaccines – which is a big deal these days. I know lots of people who are anti-vaxxers in the human realm. Those people befuddle me. So I suppose to the typical loyal consumer of anti-virus software, I must seem equally befuddling.

Caveat: Tree #920

This tree at Batan Point had a regulatory marker attached to it (i.e. a no-trespassing sign).

picture[daily log: walking, 2km]

Caveat: Fishing Report #(n+24)

Arthur, Alan and I went out fishing.

The weather was good for fishing. Mostly overcast, but only a few sprinkles of rain and the sea was utterly flat.

There were a lot of boats out fishing. I didn’t see much action on the other boats, either, though.

We did a kind of circle: Black Beach (north end of San Juan Island), San Ignacio (up and down and up again), then Tranquil Point over to Caldera Bay. We caught one fish off the middle of San Ignacio (near Silvester). We didn’t catch any at the “hot spot” from last week, off Batan Point just west of Caldera Bay.

So it was a mostly disappointing day: we seem to be back to our one-fish-a-day quota.

Year-to-date totals:

  • Coho: 11
  • Kings: 0
  • Halibut: 0
  • Other: 1
  • Too-small fish sent home to mama: 14
  • Downrigger weights left on the bottom of the sea: 1


Caveat: Tree #919

This tree is up on the hillside.

We successfully retrieved Alan from the ferry and he’s settled in here at the house. The plan is to go out fishing tomorrow.

picture[daily log: walking, 3km; driving to Hollis, 130km]

Caveat: Tree #918

This tree is reaching for the water.

The plan had been that Arthur’s brother Alan (my “other uncle”) was going to arrive at Klawock airport this evening, to stay for a 10 day visit.

Apparently, though, Arthur managed to forget to book the last leg of Alan’s journey, on Island Air Express, which is the airline that provides service on their little airplanes between Ketchikan and Klawock.

Actually, I think saying that Arthur “forgot” isn’t quite accurate: his cognitive issue is, as described before, not entirely a memory issue so much as a failure of what the psychologists call “executive function.” I see this manifest in the following way: in day-to-day experience, Arthur often “checks things off” his mental checklist before he’s done them. Thus he thinks he’s told me of a plan to go fishing, when all he ever did was intend to do so. Or he thinks he’s booked a flight for Alan on Island Air, when all he ever did was intend to do so. He plays out the plan in his mind, and his mind says, “oh, good, that’s done, then.” I think his episodic memory of recent actions mixes up “planned actions” with “completed actions.”

So Alan had no seat on Island Air, and got stuck in a motel in Ketchikan for the night. We’ll get him over to the island today, hopefully – worst case scenario, he can take the ferry in the afternoon.

picture[daily log: walking, 2.5km]

Caveat: Tree #917

This tree is another effort at trying to grow a maple tree. Along with the redwood, which I posted yesterday, I ordered a baby maple tree to make another go given my failed attempts at germination. It didn’t survive the week-long postal journey here as well as the redwood did – most of its leaves died. But it’s got a few. We’ll see how it does.

picture[daily log: walking, 1.5km; errant erranding, 4hr]

Caveat: Tree #916

This tree is a coast redwood (sequoia sempervirens). I made an effort starting a few months ago to germinate some redwood seeds, but that effort ended in abject failure. So I decided to spend a bit more money, and buy a redwood sapling, which arrived on Monday. I have transplanted it into this little bucket with some potting soil, and will keep it in the greenhouse for now. Maybe it will survive.

picture[daily log: walking, 3.5km; retailing, 6hr]

Caveat: Tree #914

This tree caught some early sun at 530, before the clouds returned.

picture[daily log: walking, 1km; currying and cobblering, 2hr]

Caveat: Tree #913

This tree on San Ignacio Island has an eagle. Can you see it? It’s very small, but clearly silhouetted against the sky.

picture[daily log: walking, 3km]

Caveat: Tree #912

This tree failed to notice the deer hiding behind a large rock. But I noticed it, and took its picture.

picture[daily log: walking, 2km; boating, 45km]

Back to Top