Caveat: rootless

Sometimes I work on my server. I have been trying to automate the map-rendering job for the geofiction site I built, that until now I’ve had to run manually. The problem I ran into comes down to permissions. Who knew that even the infamous Linux ‘root’ user is sometimes not the right person for the job? Emphasis added to the excerpt of the log file, below.

...
INFO: Total execution time: 16621 milliseconds.
Stopping renderd (via systemctl): renderd.service.
osm2pgsql version 0.95.0-dev (64 bit id space)

Using lua based tag processing pipeline with script ~/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)
Osm2pgsql failed due to ERROR: Connection to database failed: FATAL:  role "root" does not exist
...

It took me all day to figure this out. Not that I was working on it, exactly. Art and I went to town, did our shopping, came home, ate dinner. All the while, I was cogitating on this problem, and how it matched up with the results I was(n’t) seeing. And then, sitting there, it clicked.

These are the more pleasing moments of computer work – when a seemingly intractable problem presents itself and you work it through in your mind and you solve it. After it clicked, I came and opened the log file and saw the error, above, and it was an easy fix to the bash script.
picture

Caveat: HRATE

Since the school never calls me to do substitute teaching, and since it often rains outside, when I’m not working on studying history and psychology for my exams-for-credit, I continue with my activities and efforts related to my “fictional map server.”

Recently I’ve received several queries from people interested in trying to build their own “map servers.” I decided the concept needed a handy acronym, so I coined “HRATE” (High-resolution alternatives-to-earth – also, handily, an anagram for “Earth”).

I have been trying to collect in one place my documentation for how to build your own HRATE: here.
picture

Caveat: On the hypothetical value of a breakfast in the cheapest country

I've been a bit glum and very withdrawn as I confront getting ready for this unexpected trip, the situation with my uncle's health, my own feeling that a sea change of sorts is approaching in how my own life is organized… 

So I settled into an escapist weekend of map-drawing and music (mostly Mexican hiphop – go figure, right?). 

Life goes on. Wednesday, I fly to Seattle. Meanwhile, I have a vast pile of things to take care of for work.

Meanwhile, a humorous, 400 year old quote:

"Your peevish chastity is not worth a breakfast in the cheapest country." – William Shakespeare, in "Pericles, Prince of Tyre."

 [daily log: walking, 7km]

Caveat: In the mud

[This is cross-posted from my other blog.]

This isn't exactly geofiction, but I was messing around with a new project on my server.

Way back in the day (I am somewhat old), I used to play MUDs (Multi-User Dungeons). These are text-based computer games of various kinds – no graphics at all. They're a kind interactive "choose your own adventure" text, you might say. But the game mechanics in them are the ancestors of modern MMORPGs such as World of Warcraft, and there are even conceptual connections to Grand Theft Auto or Minecraft.

MUDs are quite old – they existed on mainframes at businesses and universities before PCs were even a thing, starting in the 1960s. Because of my family's connection to the local university, I was a rare child of the 1970s who actually played computer games before PCs were invented! These were MUDs and other text-based games.

When I noticed I had my own server up and running, it occured to me that I could build a MUD. Possibly, it's not even that hard. Sure enough, there is open-source software that will run a MUD on a server for you.

I chose a package called CoffeeMud. I'm still messing with it. It's very unlikely I'll ever do anything with it. But I had this domain-name, "Hellbridge.com", lying around, so I thought, sure, make a MUD.

  _    _        _  _  _            _      _
| |  | |      | || || |          (_)    | |
| |__| |  ___ | || || |__   _ __  _   __| |  __ _   ___
|  __  | / _ \| || || '_ \ | '__|| | / _` | / _` | / _ \
| |  | ||  __/| || || |_) || |   | || (_| || (_| ||  __/
|_|  |_| \___||_||_||_.__/ |_|   |_| \__,_| \__, | \___|
__/ |
Hellbridge.com                   |___/
Powered by CoffeeMud v5.9

I originally acquired the "Hellbridge.com" domain for a quite different purpose: It was intended to be a "satire-website" for a place where I once worked, which had a name where "Hellbridge" was a similar sound but darker connotation. But looking at it now, I thought it would make a great name for a MUD. So there it is.

I liked the CoffeeMud package because the admin and config of the site is mostly done from within the game. That's cool. So I create an "Archon" character, who is like God. I walk around the MUD and type "Create chair" and a chair falls from the sky. Likewise with any other object, room, monster, or character class. That's fun.

The Archon character is created in the empty, default "Fresh Install" room, by reading a book that is placed there. I read the book and I became a God.

Nice book. Note the stats jump in the prompt.

<20Hp 100m 100mv> read book
The manual glows softly, enveloping you in its magical energy.
The book vanishes out of your hands.
<1403Hp 571m 595mv>

It's not live yet. It may never be. But meanwhile, I thought it was interesting to try it out.

It's a little bit like geofiction – you're creating an imaginary world, after all.

[daily log: walking, 7km]

Caveat: A more technical summary of how I built my tileserver – part 2 (cross-post)

[The following is a direct cross-post from my other blog – just so you don't think I'm doing nothing with my free time, these days.]

The objective

I started discussing the coastline shapefile problem in first post.

Early on, I found the tool called QGIS Browser and installed it on my desktop. I used this to examine the shapefiles I was creating.

The first step was to look at the "real Earth" OpenStreetMap-provided shapefiles I was trying to emulate – the two mentioned in my previous post:
/openstreetmap-carto/data/land-polygons-split-3857/land_polygons.shp
and
/openstreetmap-carto/data/simplified-land-polygons-complete-3857/simplified_land_polygons.shp

Here are screenshots of each one.

First, land_polygons.shp

And here is simplified_land_polygons.shp

The structure is pretty straightforward, but – how do I make these? Where do they come from? – aside from the non-useful explanation found most places, which is that "OpenStreetMap generates them from the data".

The coastline problem

The way that the shapefiles are generated for OpenStreetMap are not well documented. But after looking around, I found a tool on github (a software code-sharing site) developed by one of the OpenStreetMap gods, Jochen Topf. It is called osmcoastline, which seemed to be the right way to proceed. I imagined (though I don't actually know this) that this is what's being used on the OpenStreetMap website to generate these shapefiles.

The first thing I had to do was get the osmcoastline tool working, which was not trivial, because apparently a lot of its components and prerequisites are not in their most up-to-date or compatible versions in the Ubuntu default repositories.

So for many of the most important parts, I needed to download each chunk of source code and compile them, one by one. Here is the detailed breakdown (in case someone is trying to find how to do this).

Installing osmcoastline

I followed the instructions on the github site (https://github.com/osmcode/osmcoastline), but had to compile my own version of things more than that site implied was necessary. Note that there are other prerequisites not listed below here, like git, which can be gotten via standard repositories, e.g. using apt-get on Ubuntu. What follows took about a day to figure out, with many false starts and incompatible installs, uninstalls, re-installs, as I figured out which things needed up-to-date versions and which could use the repository versions.

I have a directory called src on my user home directory on my server. So first I went there to do all this work.
cd ~/src

I added these utilities:
sudo apt-get install libboost-program-options-dev libboost-dev libprotobuf-dev protobuf-compiler libosmpbf-dev zlib1g-dev libexpat1-dev cmake libutfcpp-dev zlib1g-dev libgdal1-dev libgeos-dev sqlite3 pandoc

I got the most up-to-date version of libosmium (which did not require compile because it's just a collections of headers):
git clone https://github.com/osmcode/libosmium.git

Then I had to install protozero (and the repository version seemed incompatible, so I had to go back, uninstall, and compile my own, like this):

Git the files…
git clone https://github.com/mapbox/protozero.git
Then compile it…
cd protozero

mkdir build

cd build

cmake ..

make

ctest

sudo make install

I had to do the same for the osmium toolset:

Git the files…
git clone https://github.com/osmcode/osmium-tool.git
Then compile it…
cd osmium-tool

mkdir build

cd build

cmake ..

make

That takes care of the prerequisites. Installing in the tool itself is the same process, though:

Git the files…
git clone https://github.com/osmcode/osmcoastline.git
Then compile it…
cd osmcoastline

mkdir build

cd build

cmake ..

make

I had to test the osmcoastline tool:
./runtest.sh

Using osmcoastline for OGF data

So now I had to try it out. Bear in mind that each command line below took several hours (even days!) of trial and error before I figured out what I was doing. So what you see looks simple, but it took me a long time to figure out. In each case, after making the shapefile, I would copy it over to my desktop and look at it, using the QGIS browser tool. This helped me get an in intuitive, visual feel of what it was I was creating, and helped me understand the processes better. I'll put in screenshots of the resulting QGIS Browser shapefile preview.

To start out, I decided to use the OGF (OpenGeofiction) planet file. This was because the shapefiles were clearly being successfully generated on the site, but I didn't have access to them – so it seemed the right level of challenge to try to replicate the process. It took me a few days to figure it out. Here's what I found.

Just running the osmcoastline tool in what you might call "regular" mode (but with the right projection!) got me a set of files that looked right. Here's the command line invocation I used:
YOUR-PATH/src/osmcoastline/build/src/osmcoastline --verbose --srs=3857 --overwrite --output-lines --output-polygons=both --output-rings --output-database "YOUR-PATH/data/ogf-coastlines-split.db" "YOUR-PATH/data/ogf-planet.osm.pbf"

Then you turn the mini self-contained database file into a shapefile set using a utility called ogr2ogr (I guess part of osmium?):
ogr2ogr -f "ESRI Shapefile" land_polygons.shp ogf-coastlines-split.db land_polygons

This gives a set of four files
land_polygons.dbf

land_polygons.prj

land_polygons.shp

land_polygons.shx

Here is a view of the .shp file in the QGIS Browser. Looks good.

I copied these files into the /openstreetmap-carto/data/land-polygons-split-3857/ directory, and I tried to run renderd. This alone doesn't show the expected "ghost" of the OGF continenents, though. Clearly the simplified_land_polygons.shp are also needed.

So now I experimented, and finally got something "simplified" by running the following command line invocation (note setting of –max-points=0, which apparently prevents the fractal-like subdivision of complex shapes – technically this is not really "simplified" but the end result seemed to satisfy the osm-carto requirements):
YOUR-PATH/src/osmcoastline/build/src/osmcoastline --verbose --srs=3857 --overwrite --output-lines --output-rings --max-points=0 --output-database "YOUR-PATH/data/ogf-coastlines-unsplit.db" "YOUR-PATH/data/ogf-planet.osm.pbf"

Again, make the database file into shapefiles:
ogr2ogr -f "ESRI Shapefile" simplified_land_polygons.shp ogf-coastlines-unsplit.db land_polygons

This gives another set of four files
simplified_land_polygons.dbf

simplified_land_polygons.prj

simplified_land_polygons.shp

simplified_land_polygons.shx

And this .shp looks like this:

Now when I copied these files to the /openstreetmap-carto/data/simplified-land-polygons-complete-3857/ directory, and re-ran renderd, I got a successful ghosting of the continents in the render (no screenshot, sorry, I forgot to take one).

Using osmcoastline for my own data

Now I simply repeated the above, in every respect, but substituing my own rahet-planet.osm.pbf file for the ogf-planet.osm.pbf file above. I got the following shapefiles:

land_polygons.shp

simplified_land_polygons.shp

And these, copied to the appropriate osm-carto data directory locations, gives me the beautiful render you see now.

[Unfortunately, the leaflet script kludge that I once had working before for this typepad-hosted blog doesn't seem to be working (not allowing me to add raw javascript to my entry), so I can't replicate the slippy-map window here. But the slippy-map shows up in the original blog post that this blog post is a copy of, and of course you can see the slippy map in all its glory at test.geofictician.net (this is a test instance and may be subject to link-rot, however).]

I actually suspect this way that I did it is not the completely "right" way to do things. My main objective was to give the osm-carto shapefiles it would find satisfactory – it was not to try to reverse-engineer the actual OSM or OGF "coastline" processes.

There may be something kludgey about using the output of the second coastline run in the above two instances as the "simplified" shapefile, and this kludge might break if the Rahet or OGF planet coastlines were more complex, as they are for "Real Earth." But I'll save that problem for a future day.

A more immediate shapefile-based project would be to build north and south pole icecaps for Rahet, in parallel with the "Real Earth" Antarctic icesheets that I disabled for the current set-up. You can see where the icecaps belong – they are both sea-basins for the planet Rahet, but they are filled with glacial ice, cf. Antarctica's probably below-sea-level central basin. And the planet Mahhal (my other planet) will require immense ice caps on both poles, down to about 45° latitude, since the planet is much colder than Earth or Rahet (tropical Mahhal has a climate similar to Alaska or Norway).

Happy mapping.

Music to map by: Café Tacuba, "El Borrego."

[daily log: walking, 7km]

Caveat: Geofictician

I decided to start a separate blog on my new website.

There is a long history of me creating new "blogs" for one specific purpose or another. The longest-lived of my alternate blogs was the one I maintained for my job and students for several years. That blog still exists but it's largely dormant.

The reason for this new blog is that, although I don't mind sharing my geofiction activities here on this blog, I'm not sure how open I want to be about the rest of my life with fellow members of the geofiction community where I participate. That is, do they want to see or do they care to see my poetry, my ruminations of day-to-day classroom life, my oddball videos and proverb decipherments? 

Since I think it's better to keep those things separate, I decided to make a separate blog. I also did it just to support the "technical unity" (if you will) of the website I've been constructing. 

I may develop a habit of allowing the things I post on that other blog to appear here, but not vice versa. This blog would be the comprehensive "all Jared" blog, while that would be a kind of filtered version for the geofiction community. 

Anyway, here's the blog (blog.geofictician.net), which currently has 4 posts, created over the weekend. Note that it seems like this blog will be fairly technical, representing the most abstruse aspects of my bizarre and embarrassing hobby, which might be termed "computational geofiction."

[daily log: walking, 7km]

Caveat: seasonal drift…

I don't have much to share. I've been a bit out of sorts and disgruntled, lately, and it would be hard to pinpoint a single, specific reason.

Partly, I'm feeling a bit stale with work – I need to do something innovative, but in the current class schedule, I feel I have quite limited oportunities for that. Because of changes in the way the 7th grade (1st year middle school) exams are set up in the public schools, Karma has adapted by NOT offering a special exam prep schedule for 7th graders. The consequence of that is that I no longer see much of a "Naesin vacation" as I used to call it: that is, a seasonal slowdown of the teaching schedule, including Saturdays off, during the test-prep period. Now, instead, it's just the same-old same-old. So it leads to a feeling of burn-out with respect to work.

I have a certain hobby I don't post very often about: the geofiction thing. I've shared it a few times here – it's not top secret, it's just something I figure most people don't find particularly relatable. Mostly, it involves drawing fictional maps in a digital environment. However, as I mention in the lefthand column of this here blog thingy™, I also have taken on a certain level of administrative responsibility for the website, on a volunteer basis. Lately, that has been profoundly unrewarding, due to some unpleasant personalities on the website. The consequence is that I have scaled back my participation in the website, and I suppose that "loss" is also contributing to my current sense of disgruntlement.

Of course, I always get a bit melancholy around the equinoxes, too. That's an inexplicable and perhaps untypical manifestation of some kind of seasonal affective affliction. It's as if it's not the presence or absence of daylight that gets me gloomy, but rather periods when the amount of daylight is shifting rapidly, in either direction.

Anyway.

[daily log: walking, 7km]

Caveat: Tárrases



I’m not exactly in the closet about my geofiction hobby – I’ve blogged about it once or twice before, and in fact I link to it in my blog’s sidebar, too – so alert blog-readers will have known it is something I do.

Nevertheless, I’ve always felt oddly reticent about broadcasting this hobby too actively. It’s a “strange” hobby in many people’s minds, and many aren’t sure what to make of it. Many who hear of it percieve it to be perhaps a bit childish, or at the least unserious. It’s not a “real” hobby, neither artistic, like writing or drawing, nor technical, like coding or building databases. Yet geofiction, as a hobby, involves some of all of those skills: writing, drawing, coding and database-building.

Shortly after my cancer surgery, I discovered the website called OpenGeofiction (“OGF”). It uses open source tools related to the OpenStreetmap project to allow users to pursue their geofiction hobby in a community of similar people, and “publish” their geofictions (both maps and encyclopedic compositions) online.

Early last year, I became one of the volunteer administrators for the website. In fact, much of what you see on the “wiki” side of the OGF website is my work (including the wiki’s main page, where the current “featured article” is also mine), or at the least, my collaboration with other “power users” at the site. I guess I enjoy this work, even though my online people skills are not always great. Certainly, I have appreciated the way that some of my skills related to my last career, in database design and business systems analysis, have proven useful in the context of a hobby. It means that if I ever need to return to that former career, I now have additional skills in the areas of GIS (geographic information systems) and wiki deployment.

Given how much time I’ve been spending on this hobby, lately, I have been feeling like my silence about it on my blog was becoming inappropriate, if my blog is truly meant to reflect “who I am.”

So here is a snapshot of what I’ve been working on. It’s a small island city-state, at high latitudes in the Southern Hemisphere, with both “real-world” hispanic and fully fictional cultural elements. Its name is Tárrases, on the OGF world map here.

Here is a “zoomable and slidable” map window, linked to the area I’ve been creating, made using the leaflet tool.

There were some interesting technical challenges to get this to display correctly on my blog, involving several hours of research and coding trial and error. If anyone is interested in how to get the javascript-based leaflet map extension to work on a webpage (with either real or imaginary map links), including blogs such as typepad that don’t support it with a native plugin, I’m happy to help.

I have made a topo layer, too. I am one of only 2-3 users on the OGF website to attempt this – But the result is quite pleasing.

I have always loved maps, and since childhood, I have sometimes spent time drawing maps of imaginary places. However, I never dreamed that I’d be producing professional-quality, internet-accessible maps of imaginary places. I believe it is a kind of artform.

So that’s where my time off sometimes disappears to.

UPDATE NOTE 1, 2016-12-05: The topo view is currently broken due to some work I’m doing. It will be repaired eventually.
UPDATE NOTE 2, 2017-02-16: The topo view has been repaired.
UPDATE NOTE 3, 2019-08-15: I noticed while doing other blog maintenance that the leaflet embeds were broken. I spent a few hours fixing them – apparently some recent leaflet.js update wasn’t backward-compatible (argh).

[daily log: walking, 1.5km]

Caveat: an obsessive tinkerer tinkers obsessively…

At the risk of becoming boring, posting on the same essentially autobiographical topic for the third day in a row…

I continue to obsessively mess around with my computer, trying to figure out what happened to it. There is a component of my personality that is a compulsive tinkerer, and thus I somehow prefer to try to fix a clearly dying computer to buying a new one. I suppose partly I see it as an opportunity to "prove myself" and make sure I possess at least some of the skills necessary to be "self-sufficient" in the context of computers.

I made a very weird thing happen: when I gave my computer a complete "cold" shutdown (i.e. I removed the onboard battery, which forces the BIOS to reset), my USB bus returned to life! This seems quite weird and miraculous, but I can just barely grasp how this might work. If something happened that had caused my BIOS to break, which had in turn been the cause of the lost USB bus, by forcing the reset I recovered the original BIOS configuration.

Well, anyway, in theory this means my computer isn't actually broken, at the moment. But I have lost my trust in my computer – I'm working hard to make sure nothing would be lost if it should crash catastrophically. This is a useful exercise, which I don't resent.

I continue to tinker with Linux – it's interesting to me, at an almost obsessive level. I'm curious, now, to see if I can replicate ALL the functions I was performing on my home Windows machine – because my relationship with Windows was always a marriage not of love but of convenience. I had concluded 4 years ago that I could NOT replicate all those functions, but having solved the language issue yesterday, I feel optimistic that Ubuntu has progressed to the point where I maybe can do it.

There are some challenges:

  • getting my massive music collection (18000 tracks? – I didn't even know!) to be accessible and playable – every time I try to configure one of ubuntu's music players and point it to my music collection, it crashes;
  • configuring my  offline mapping tool (JOSM) that use for my geofiction hobby; this should be easy, since JOSM was originally written for Linux, but I'm running into problems;
  • replicating my "sandbox" database (postgresql) and coding environment (perl / python) – because I have a mostly dormant hobby of trying to keep my programming skills functional, in case this "teach English in Korea" gig falls apart, or if that worst-case-scenario related to my mouth health situation eventuates, and I experience a major impairment or loss of my ability to talk.

[daily log: walking, 7km]

Caveat: A holiday in Tlön, via Khaiwoon

I have a hobby I don't talk about much – because it is not something most people can understand, and I don't always want to try to explain it. It does not even have a single term that describes it, but probably the most commonly used these days is "conworlding". This is derived from the noun "conworld" which is a contraction of "constructed world".

I have been doing it since childhood, when I called it "drawing maps", because that is how it started: drawing maps of imaginary countries. But by my teenage years it had become "writing encyclopedia articles about imaginary places." Mostly, I would fill notebooks with this material, as I was never satisfied (or expert enough) with the graphics software available for drawing maps, so I always drew the maps by hand.

This type of activity has a respectable side: JRR Tolkien apparently drew his maps and wrote his appendices for Middle Earth long before he wrote his novels. He also took very seriously the related pasttime of "conlanging" (inventing imaginary languages – not that he called it that, as the term came later). His complex Elvish and other languages are serious philological works. More recently, serious "professional" conlangers have even been able to make money: the guy who invented the Klingon language for Star Trek got paid something, and there's someone who works full time as a conlanger for the Game of Thrones TV show. I was always too perfectionistic, due to my linguistics background, to go very far with conlanging. Arguably, the the same thing that challenges me in learning Korean is what prevents me from being a serious conlanger. Setting that aside, however, I love to make maps and craft the fictional geographical data that accompanies them.

I remember when the wikipedia first appeared, I thought, "there should be a wikipedia for fictional places." Actually, this was an echo, updated for the internet age, of the themes in Borges' famous story "Tlön, Uqbar, Orbis Tertius." I was so enamored of this idea that about a decade ago, I tried to use my database programming expertise to build a fictional online encyclopedia for myself, but that project lost momentum at some point.

Well, it turns out some guy in Germany has done it. Further, there is a google-earth-quality online mapping tool for it. Actually, the mapping tool was the thing created first, and I found it about 6 months ago – the wiki came just in the last few months. There are people who are much more serious about it than I am, creating fictional countries, with supporting encyclopedic articles, that are difficult to distinguish from reality. Tlön, indeed.

The collaborative aspect is what is genuinely new, and it changes things some. It is interesting to see what other people do. So I have been spending some time there – mostly crafting maps for my allotments (one can sign up for free, and receive a "country" by request – a terra incognita to do with what one wishes), but also writing some wiki entries for them. It has the same appeal to me of sim games like simcity or civilization, which I have spent plenty of time addicted to, but it is better: there are fewer rules, and the result is always cummulative and feels more creative.

I have attempted to attach a screenshot of a map of the downtown of the city state of Khaiwoon, a vaguely Singaporean nation created by one of the most prolific and talented conworlders. The site is called Opengeofiction – here is a link to his article about Khaiwoon. There are links from there to the mapping tool.

1366px_khaiwoon

I will leave as an exercise for curious readers to find which countries are "mine".

[daily log: walking, 1 km]