vanderwal.net

Eight Rounds of Link Clean-up

I completed this round of links clean-up of my links page, which was the eighth pass. I went from 943 links on the page to 866 after removals. I have added a few new links from my usual laissez-faire finding methods.

The process honed into the Python link checker outputing a markdown file with sections for: Broken; Blocked or access denied; Timed out; SSL / certificate errors; Connection errors; Working, but redirected. I have the script put the related links that have been flagged into tables in these sections.

After the fifth or sixth pass I added an additional column to the tables in the markdown output note for “Action Decisions” where I put my annotation for: Keep; Delete; new URL (with the link); and update (for the redirected links section). After this eighth pass I added a small table at the top of the markdown output from the Python script link checker with each of these four decisions at the top. I had a couple of days between the sixth run of the check and the seventh and eighth (the last check was the post final clean-up check). Going through what remained on the seventh pass (mostly the 90 redirections) I swore I had more decision action types I had used, so having the list in the note I use for my manual assessment seemed really helpful.

This Task was a Sad Task

I’ve run a version of this links page for about 30 years. It has largely been an act of accretion. There have been many links that have gone dead and things no longer living on the web (or no longer living period). My link checker has done a decent job until about three or four years ago when my local version of Python didn’t mesh well with the older BeautifulSoup package. A couple of years ago when I got them in alignment there were quite a few false positives. But the task also had many links that had gone away.

This current update to approach (see Links Weren’t Linking :: Off the Top :: vanderwal.net) drastically improved link checking, but also pointed to a fragile state of the web. Not only the cause of massively increased friction to run a link checker (a couple of orders of magnitude increase in crawler and bot traffic that is non-organic and non-productive). This traffic isn’t out of malice, but ignorance, poor thinking, and poor planning (a common aggregation that is a central consumable to build from is better than thousands of rogue entities).

I have many “archive” links that I grey out where the site is mostly functioning still, but not been updated in years. Many of these I found in this current run are now completely gone.

There are many personal sites and blogs gone. But, also many subject focussed sites in urban planning & cities focussed that are gone, along with many in the Food & Coffee & Travel section, Productivity / PKM / Systems, and the Enterprise / KM / Learning sections. So many good resources, mostly from individuals who are crafting, honing, and adding incredibly good and smart insights that are now gone.

Some of the redirects had gone from personal insights to machine-generated content with factual errors in abundance or ad driven content, which are pretty much indistinguishable from the former.

I’m Looking to Replace Lost Sources with Good Human Content

I started blogging in 2001 to put things out into the world to have conversation and share things from my personal world to my work world. I found blogging is an utterly insanely good way of finding people of like minds and interests. I found the fellow curious travelers who wanted to learn more and understand the small to the vast interconnections of systems, cultures, and diversities of understanding in this wonderful world of “other possibilities” that surrounds us. I write and share only what I personally create out of my mind to understand what I think and find cracks that inspire more curiosity. The nodes on the web that are blogs worked through good search, which is now mostly gone as Google seems to have lost interest and there isn’t really a viable replacement for that widening gap.

I’m deeply looking for replacements in the sections where I lost sites that provided good solid insights and sparked not only understanding but fact-based inspirations to understand more and incite more exploration.

Many of the people and voices that over the years drifted off from posting to their own sites I deeply miss. Some have drifted to silos that are (have become) horrible places for sharing information as they are really difficult to do anything positive with that information and what has been shared. Decent town square sites / services and plazas for people to digitally gather and share and interact have deteriorated and/or been manufactured by those running them to be problematic spaces by intent. Some I don’t go to at all, some I wander through, and some I wish their leadership cared (or had an inkling of understanding) to get back to something that was much better.

My Links Page is Largely in Decent Shape

My links pages has long been my loose tethering to others and other web spots that had information, people, and ideas of interest. It is for thin strands of interest as a weak attractor in a Model of Attraction sense. I can click a link and attract something to my screen of interest. People and their thoughts are drawn closer. These links are now mostly working after cleaning up more than 400 of them.

Links Weren't Linking

For more than a decade I have had a Python link checker I occasionally run against my Links :: vanderwal.net page. I regularly update the links page and the additions are manual inserts and reposting the page when I find something I’m regularly going back to, or want to go back to. About 18 months back my long used python script using Beautiful Soup (HTML parser) - Wikipedia needed updating the BeautifulSoup package and tweaking the script a little bit.

Over these last 18 months the bot / scraper / AI crawler wars have become a hot mess and running a link checker script, unknowingly was getting blocked by hosts / servers that are working to not get slammed by the orders of magnitude of increased traffic from non-organic sources. One method (among many) server owners deplay is blocking access to websites from requests with no user agent or an old user agent claim (as this is what a lot of the aggressive inorganic traffic uses). My old school link checker feels like a web version of Wall-E going around and checking to clean up my broken links and getting caught in the wake from large AI industrial retrieval ships.

Thursday I wanted to run my link checker and realized it wasn’t going to be all that helpful. I thought a quick edit of the script to add a more current user agent to the what is presented to the link’s server would help. I went with: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 ” “(KHTML, like Gecko) Version/18.0 Safari/605.1.15”. This one minor change when running the link checker removed about 200 false positive broken links.

There are How Many Links?

With many things I have run and been doing for ages, my estimation of how much or how many of something is wildly off. Another minor tweak I made was to list the number of links tested on my Links :: vanderwal.net page. I have had a links page for two or three years longer than I have had a web domain. I set up this domain in 1997 and the site started with a homepage and a links page of things I wanted near in reach. My first link check I made on Thursday I had 943 links. Well, that was about double of my rough estimate.

Updating to Get Better Data

When I last updated the Python script to a new version of BeautifulSoup I didn’t really change much else. Now that things were less than optimal I noticed my Python script just used a default check, but wasn’t using GET. Adding GET would provide feedback as to the type of HTTP response codes I was getting and that would help reduce my manual checks of the list of “broken links”.

A quick test after adding GET had more updates and parsing needed. If I’m getting the actual responses back, I want to log the returned codes and information, but I really want that organized. The initial output of the checking is a long list of links with non–200 status with their state, the link, error code, if a redirect it tried the redirect and if a 200 it shows the “200” and the new link.

The initial output from running BeautifulSoup is the list in a temporary text document. The last part of the script runs through the text document and turns that list into a grouped list and table markdown file in my Notes directory that Obsidian watches over.

Structured Documents Made the Next Steps Easier

The structured Markdown note has sub-headings for the different returned link states. This works like a really good triage list. I could easily double-check the links marked as broken (404, 409, 500, and no DNS) in the table and added another column to the table for the checked status I provided. I had 52 of these and I went through them in about 15 minutes.

I made a back-up of the links page and verified git was current so I could rollback. I handed off the update to Claude Code to follow my notation of the section I just checked, my structured outcome note, and to follow one of the four processes I posted. It made the updates in seconds and I ran diff against the two files and the outcome was perfect.

I moved the outcome page to production and ran a commit. I ran the Python checker again and tackled a couple of small sections in the same manner. On my 3rd running of the checker I looked at the roughly 430 redirections captured. Since Claude had managed the two prior updates perfectly I had it handle the redirection updates. This too was processed quickly and the output links page was perfect on the manual check. But Claude Code running Opus 5.5 had additional notes from the update that included some redirects were not similar to the page historically that I linked to (in some cases the domain had changed hands years ago and Claude derived context from surrounding links that it was no longer a fit), noted that sites had changed languages, sites were now ads from domain resellers, and more. It had a list of links that could now be problematic. I wasn’t expecting this additional feedback and the feedback I checked was spot on and insanely helpful.

I ran the Python link checker a fourth time and I was down to 916 links on the page. Broken links gone, with many of my “archive” links gone too, which are the gray links as the linked site went from not being updated in years to now fully gone from the living internet. I have left things with this fourth pass of the checker on my to-do list. The links that are large sites with large organizations behind them are in the blocked connection or have “are you human” test in front of them, but still function. I have about 70 or so links that I need to manually go through and log what to do. Some no longer have the content I have long pointed to.

Outcomes

I’m happy with where the Python link checker ended up. The web is a lot more fragile than it seems like it ever was. I was surprised how many links had redirects (I have 50 that still need manual reviews). While I’m bummed many “archive” links are not on the page they were gone and they could be let go.

The two sections on the links page that took the biggest hit are the tech section and the “Urban Planning and Cities” section. Finding sites that fit the urban planning and cities perspective have always been tough to find. Losing some means I need to dig again.

Rearranging Automations and Aggregations

This week and the prior had a spattering of updates and reworking of some of my regular automations and aggregations.

I’ve posted about my automation patterns with scripts (mostly python), some AI (sometimes Claude), many web scripts and web services (IFTTT and a few others), many local scripts and triggers that one action for an intended purpose also pushes things to additional places (for aggregation, organization, and later retrieval), other patterns. I’ve written about some of this recently: My Automations Over the Ages and Today’s View and Adjusting Daily Aggregation for Less Friction.

I’ve long played with data to sort out what is in it, the shape of it, and possibilities in it (going back to before grad school – it was a large part of my second job out of undergrad and grad school expanded and honed it). This continues today with different tools and services. The ability to quickly build a proof of concept for data analytics and munge the data to get a helpful data visualization is something I’ve liked from AI services like Claude. I can test a workflow, set up a quick process for making use of the data and information I normally tuck away. Most often I’ll try something and if it works well I will modify it and move it out of Claude and into a python script (or set of python-based processes) to get a usable output I can gain insight from and/or work with. On occasion the data is not steady enough, so I will keep things in Claude and run scheduled automation to do the grunt work so that I have my aggregations in my Daily Dump markdown note I keep in my notes folder Obsidian looks over, and it is ready early in the morning for me to sort out what I continue with, what is fine, and what I need to process in some way.

Nearly all of my morning aggregations and automations are dealing with local (on my device(s)) data, information, and objects. I don’t want this going into anybody else’s service or cloud as it is foolish and pointless to do this. This is my Personal InfoCloud (see Overview of the InfoClouds – Personal InfoCloud) and it remains there not drifts out.

Claude Cowork Shift Local Scheduled Automations

A little less than a third of my morning automations and aggregations have run through a local Claude Cowork Scheduler. Anthropic announced it was not going to allow the creation of new Cowork Scheduled tasks and new ones (possibly modifications too) would need to move to the cloud. Uh… no! The options were to move fully to Python and/or other scripted workflows, move off of Claude, or move the scheduled tasks out of Cowork to Claude Code. I also had a few morning automation and aggregation tasks I kick off manually as they are based on me finishing my personal tasks.

Manually Mapping Out Flows in Obsidian Canvas Helps

A little more than a year ago I mapped out my morning and weekly workflow automations and aggregations with the services, scripts, tasks, time triggers, schedulers, and regular manually triggered tasks, automations, and aggregations. I mapped this rather quickly in Obsidian’s Canvas Obsidian Canvas that is helpful for seeing groupings, related processes through links, and flows. I went through my grouped tasks that I automated and marked the ones I needed to deal with using a yellow dot emoji. There were a few nested ones I hadn’t considered. (A great thing with Canvas is having a link to my notes in the automation node so I can quickly jump to see details.)

I have long used Claude Code for debugging code and suggesting optimizations to code (sometimes the suggestions are good and quite helpful, I moved to it after deep frustration with ChatGPT and GitHub Copilot poor output). Moving things to Code was relatively straight forward (it didn’t have the battles with time that were prevalent in Cowork, which was really frustrating until I sorted out a workaround so Cowork could understand “today”, “yesterday”, “past 24 hours”, etc – working with time can be problematic, but it mangled things that are relatively straight forward).

Move from Claude to Python Driven App, Triggered by Apple Shortcuts

I went to convert one of my manually triggered automations and realized it didn’t need to be in Claude at all, and was a good candidate to move python, but I would need to build a Slack app / bot to pull from my personal Slack instance. About 5 to 10 years ago building a Slack app was rather a common and straightforward endeavour. That changed a bit and the setup screens did too. Slack is now AI bot-friendly first and I’m not building an AI bot, but going in the opposite direction (I had used the MCP from Slack to run the aggregation prior). Rather than 15 to 30 minutes creating the Slack app it took around 90 minutes to deal with the changes in Slack. But, I also wanted an easier way to trigger the Slack app to run and capture the links to videos I had found interesting, and put them in a list in a markdown file, so I could take the next steps. I ended up using Apple Shortcut to trigger the Python script. The automation now is done in under a second rather than just under two minutes and no files uploaded anywhere.

I have two other automations I manually trigger that I can likely move to Python in the same manner using secure APIs and triggering them from Apple Shortcuts. I haven’t converted them yet.

The Shift to Code Started a Few Months Ago

In the past few months I moved a few of my Claude tasks I run for clean up of transcripts from Cowork to Code and found a large reduction in time, tokens, and Claude usage in that move. The quality of the shift has been better quality, output is more consistent, and also calls out potential issues on transcription clean-up (I pull podcast and YouTube transcripts which are riddled with errors and poorly structured, I don’t change the content but have proper sentence and paragraph structures, but also included some sub-headers). Some of the data conversion tasks and modifying script tasks that I would run or plan out in Chat or Cowork all moved to Code and the quality improved and token / usage all noticeably improved. The usage and quality improved when I started using Opus 5.5, with Sonar I had a lot of rework and Claude off and running changing things that were reviews not to change things. My token and usage decrementing is about a third less or better with Code and Opus 5.5, which doesn’t fully make sense.

The last push was moving my Daily Dump related aggregations and automations and the Weekly round-up out of Cowork as everything done is local with those tasks. The move from Cowork to Code was rather straightforward. I asked Claude Code to guide setting up the needed resource access it would need for its own processing and assistance setting up the Routines and its timing (Code’s routine timers aren’t precise and I wanted them running at the exact open times with locked windows I regularly run things within). That timing took had pushback from Claude but got it close to the window and then it took me understanding Code Routines have some odd buffers of a few minutes.

How it is Going

This took a chunk of time early in the week, but things are running fine. All but a couple of manually triggered tasks are now green check emoji and not yellow dots. Tomorrow morning when my Weekly roundup runs will be a big question and test.

The biggest friction points are the pieces I’ve moved to new homes and remembering where they get triggered. The upside with using Apple Shortcuts is that it can be organized rather well to keep things related together. I moved a lot of my local web servers that enable data visualization web apps to Apple Shortcut triggers while setting up other Shortcuts and that has helped the efficiency of those processes.

I Wrote What?

Part of the impetus for writing this post has been driven by the updates at A new Observable | Observable, which I recently wrote about Large Observable Update :: Off the Top :: vanderwal.net, and looking at changes in not only Observable, but updates to D3 by Observable. These current and not-too-distant updates have had me thinking about what I could do with some of my blogging data to make interactive charts and graphs. I’ve recently moved a lot of my legacy static charts and graphs to Plotly to gain interaction with them, for many of my own uses. But, in the process of getting a decent data workflow setup for my site (and my notes data) I have relatively clean data locally ready for use and testing.

A Little Data Workflow Background

For this blog I have been pulling down the data from the site’s backend and converting it with a set python script to extract it to SQLite, which is relational but quite fast for reading data and is great for single user write and a few read options. Over the past few months I’ve also been building many data analytics and data visualization prep python scripts (some building JSON files for standalone webpage interfaces). I’ve used Claude to debug, but also just turn out a script in less than couple minutes for the page setup for the D3 or Plotly scripts, which would usually take me 30 to 120 minutes. Once the scripts are set and I’ve reviewed them it is all just setting python flags and minor tweaks to the script to handle a data update.

With these script setups I’ve been able to try quite a few D3 chart and graph options very quickly. Going through the Python code, the data structures for the visualizations, and the hooks for the visualization and I’m rather happy with it for quick outputs.

What Can I See Using Blaugust Data

One of the things I had interest in from my A Wrap to Blaugust / Blaugust run in August was what did all of that look like from a basic statistic and graphical perspective. A few months back I created and ran a bunch of Jupyter queries and visualizations, Personal Blog Data Analysis - Looking at 25 Years. Since then I’ve been thinking about looking at a monthly view with more interactive charts and graphs. Since I had a decent amount of data from August 2026, existing data workflows, and a few D3 charting and graphing options I could work with I took a look.

August Off the Top Force Graph

The first is an interactive forced graph with the blogpost as orange nodes with the size corresponding to word count and the tethered edges link to categories for the post. Clicking on the post will show the related categories and pulling on the post node will start to see the other tethered posts connected through the categories. There are few toggles: 1) Posts and categories or just Posts with their related interconnections; 2) Only show shared categories (a category with more than one post); 3) Post to Time, which may be my favorite view, as it take the graph and lays out the posts left (oldest) to right (most recent), with a small light timeline at the bottom and all the other relationships displayed as well; 4) Post titles, which clutters things a bit and adding the toggle resolved that clutter.

You can view this standalone interactive web chart that sits in the Lab :: Off the Top Analytics at the direct link: Off the Top, August Force Graph with Toggles. A sample is here:

August 2026 Word Counts

As I was going through the data for August 2026 I knew and could see the word counts for posts, which I don’t have easily (nor quickly) available on my site. But, with the data local in SQLite and I’ve been using the local data to run many types of post analytics, word counts become a simple and quick option that doesn’t impact the site’s performance at all.

I was really curious about word counts per post across August, but also was more curious about a cumulative word count across the month as well. A quick and easy bar chart and line graph solved this, both having interactive hovers for metadata. August had 25,390 words posted.

The standalone interactive web chart is also in the Lab and the direct link is: Off the Top, August Word Counts by Post and Cumulative Count. A sample is here:

Cumulative Word Count for 2001 to 2026

Looking at the cumulative count for August 2026 (the 25,390 words posted in the month) I had no idea what that count was in comparison to the whole life span of the blog (under this CMS that started in late October 2001 and not including the static posts prior to that). I know that in the early months and years of the blog I wrote a lot of short posts and the posts were frequent. Looking at word count I wasn’t expecting the first two or three years to have roughly the same growth slope up to 2008.

Looking across 25 years of posts I knew a single post analysis was going to be a bit rough for an interactive standalone webpage. Monthly would be better, but could still be a bit too much to process dynamically in the browser. Quarterly, was just a bit on the thin side and more difficult to see trend shifts and correlative line slope. I settled on a two-month interval and I had something interesting in under a minute.

July and August 2026 were the highest word count across all time on this blog. The 25 thousand words is roughly a quarter of a 250 page book. But, now that I had the bi-monthly chart I can also see the early years I had roughly similar counts per two months up until 2008 and then there was the “great flattening”, until the “great pandemica” lockdown when things ticked up again until lockdown ease and work went back to its normal patterns.

I ended up over the 25 years with 437,417 cumulative words just on this blog. That is roughly 4 books of 250 pages. It would be a rather odd book, but in just considering word count as the focus. I find word counts odd, particularly writing to word counts. I’ve had a regular column in tech publication that had a 750 rough word count topic, this was at a time when my posts were often 1,000 to 1,250 words on average and 750 words was about where I had the framing set to get to the meat of things. I mostly write on this blog to an amount where there aren’t good words left, which is 200 at times to 2,500 or more. To see my writing shifts in the early years roughly similar in trajectory rather interesting.

The standalone interactive web chart is also in the Lab and its direct link is: Off the Top All-time Cumulative Word Count. A sample is here:

Will this Continue?

Now that these are standalone webpage charts and graphs, I’m not sure what I will do. I need to get the templates consistent from a layout and design perspective. But, once the standalone webpage is created I just copy it to the Lab or other location, and this can be automated with a quick script. I think the month-by-month analytic is interesting, at least to me. I have not only the local standalone version that I posted, but a rough output build that isn’t fully self-contained that I can play with and iterate on.

If you like this and want to see more ping me on one of the services or send an e-mail using the Contact in the navbar.

Large Observable Update

Around mid-week Observable announced a major update on their blog (A new Observable) and my evenings turned into looking at updates to the Notebooks, charts / data visualization offerings and much more. There are more interactive charting models and this is the first time I’ve noticed a downloadable notebook app for Macs (I haven’t tried it yet).

I use D3 by Observable quite a bit for data visualizations form data focussed apps (I also have been using Plotly for local Python interactive data apps). I started using D3 in late 2011 for some small data visualization projects, but in 2012 was working on a large project that made extensive use of it and it has stuck as a tool in my tool belt.

I often use Pandas, Polars, and Jupyter, but also RStudio. It might be time to look at Observable’s notebook, but I’m not overly fond of JavaScript and I’ve been using Claude to assist cut down the struggles with JS and getting to looking at and interacting with data visualizations.

The D3 and Observable sites have been some of the best resources for understanding data visualization basics and good fits for data and visualizations. The education aspect, as well as the really good quality charts and visualization tools have made it a regular resource that I use and regularly point others to.

I still have a lot of Observable and D3js tabs open as I’m pulling better notes for myself. But, I figured it was a good time to share it with others.

Adjusting Daily Aggregation for Less Friction

Sunday I hit a point where my daily automations (which I layout in part here - My Automations Over the Ages and Today’s View in my morning process and weekly process) and placing the outputs into my Daily Dump note sections was really making for a lot of friction in my note taking / note making. Additionally, it was breaking focus and holding my concentration on what I was trying to capture in my note’s sections.

How Daily Dump Notes are Setup

My Daily Dump has 14 sections: Plans, Thoughts, Review, Read, Talked With, Listened, Watched, Productivity, Worked, Learned, Played, Ate, Health, Bought, and Added to a wishlist.

The intent with the subheads in my note is to quickly jump to the section as I run across something or am thinking through and idea or concept and capture the thoughts or what I consumed and ideas from it, so I can get back to it later in my notes. But, since this is markdown and in Obsidian I can take the notes with its subheading and text, extract it, and then transclude it (move it out and link it back in to show it in another note). But, having each day’s note relatively clean (the exception is the Plans section for things brewing and on the front burner to do are moved from the prior day’s note to the current day’s with a python script that keeps the completed tasks on the prior day and removes things not completed and moved them to today).

Friction from Adding Aggregation to the Sections

The daily aggregations pulled in things captured from various reading and copying or favoriting tasks that get automatically captured in Instapaper and Pinboard, as well as all the files I created and modified. I also have notes from Drafts that I capture on the go and tagged to get pulled in aggregated, as well as a list of things I watched online that I want to track, but also list of videos watched where I’ve captured the transcript in Readwise I may want to review (and likely do a further clean-up step).

All of this was getting aggregated into collective notes by type and transcluded into the relevant groups in my Daily Dump note. This added a lot of friction to the intent of my Daily Dump for quick easy drop into a section and capture a note and then after sort out how I may handle it. Add to this, Sunday my Weekly Review runs and captures all of this from the week and compiles it into one note so with a heat map at the top so I can review it to see what I’ve been doing and find things to cut back on or put focus on for the upcoming week. The Weekly Review was transcluded into Productivity, which is ironic as all of this adding of aggregation transclusions was making me really unproductive.

The Fix to Bring Back Focus and Quick Capture

On Sunday I had enough of the friction and took time to sort out a solution. I liked the capture and the ability to scan through and find things I need to continue working on or make a change on, but having it in my quick capture notes area made it tough.

I added a “Review” section just under Plans and now all aggregations and transclusions go there. The best part is when I’m not reviewing it, a heading in Obsidian can be collapsed, which gives me focus on everything else.

On Sunday I did a quick manual restructuring of my note by putting everything aggregated into the Review section and then collapsed it. This was a magical focus trick.

Updating the Python Script that Automates Transclusion

Sunday I did the first edit of the python script that takes the aggregations notes created earlier in the moring and transcludes them in and repointed them to not go into the other categories, but all go into Review. This hasn’t run perfectly yesterday nor today (it works when I trigger it to run manually, but not from the trigger to run the script), but it is getting closer.

Protecting Focus is Key

Protecting focus and being able to quickly shift focus and be productive is key and has been something I’ve been working to do since I was a kid. Over the years I’ve learned the distractions and how to pull focus and calm back in. Removing the friction from abundance of information already in where I needed clean space to quickly capture created the large problem.

Now my big next step is sorting out the time to actually get a good weekly review done from what I’ve aggregated. The capture has been really good, but I just need to get better about protecting time on Sunday (or move this to Saturday) so I can sit and focus on what was done, what I can reduce and what needs focus to get done.

My Automations Over the Ages and Today’s View

For a few decades I’ve converted regular manual tasks into automated tasks for many things. I use short code keyboard conversions with TextExpander (TextExpander), Alfred (Alfred), Keyboard Maestro (Keyboard Maestro), Hazel (Hazel by Noodlesoft), Apple Shortcuts, and a couple other small apps and services. But, I also have quite a few scripts in python and Ruby, but a few pure bash scripts. I have a handful of apps that use Claude Cowork (Claude Cowork | Claude by Anthropic) on a schedule and used where the data is not always standard, so it aggregates and structures the data where it often later gets processed by python.

I started building and running automation scripts in the 1990s to handle tedious tasks with multiple steps, or would likely be victims for typos, and which I was handling daily. A task that would take 3 or more minutes (often longer) and had structured stable data I could search made it viable to become a script and automated. I had them to filter email to folders, filter downloads to common folders / directories, and one to turn off audio for the evening which monitored Twitter every 30 minutes for my nightly signoff (“off to read and sleep - zzzz”) and it would mute audio. I’ve had a few for audio that would capture podcasts I was on for a local copy during recording that took just one button click to set Audio Hijack Pro properly and it just worked.

Moving to the Current Adaptive Flows

The last six months (or a little more) I’ve been building aggregators to capture what I’ve done the day before (and the week prior for a manual review). I’ve also automated my morning Obsidian (Obsidian) setup, so that my daily note (or Daily Dump for my notes, in my vocabulary) templates are created for the week ahead with a python script, rather than creating a new note and use a TextExpander snippet to build the structure each morning. But, doing this I realized that I could take my daily task list and the list of potential tasks I keep at the top of my Daily Dump and script the moving uncompleted items to that day’s Daily Dump from the day prior, but also quickly run a calendar scan for the day with Claude and the BusyCal MCP to have my known items at the top of the note written directly in the note. The completed items and checked off were left on the prior day’s Daily Dump in their sections.

In order to run the scheduled python scripts I need my laptop to be awake in the background (but it still the screen remain locked) so to run early morning tasks. I found PMSET as the key to handle the wake-up for the first run and have a python script handle the remaining wake-up processes for that day.

I now have a mix of Claude Cowork Skills (which if I sort out the data is stable enough I convert them to python for consistency, speed, and sustainability) and python scripts that pull my prior day’s files created and worked on list, my items in a feed of the prior day’s additions to Instapaper, Pinboard, and a list of new items in DevonThink as PDF (with links to the PDF as well as a working URL to the source of the item), a list of Readwise articles and video transcripts, and videos I liked and sent to my personal instance of Slack. These lists are in markdown and placed into the directory Obsidian watches over. Most of them a little later in the morning are transcluded into my Daily Dump note for the day where I read through them.

Morning Process

My morning process is to read through what was done the day before and edit them to add links to other resources in Obsidian and write notes. Often I have some entries that will turn into that day’s list of things to do or one of the “potentials” list. I will take items that need to be their own item note or be included into a concept note. This keeps the items fresh on my brain and connecting things to where I can get back to them more easily.

I write this today, as this morning I found my morning automation hadn’t fully run properly, because my Sunday evening script to create the week’s Daily Dump notes didn’t run as it was a cronjob and I was cooking with the laptop asleep so the job didn’t kick off. This morning I converted the cronjob to a launchd task instead and to run on Sunday during another automated wake open window.

Weekly Aggregation

Sunday’s I also run a weekly wrap-up, that pulls everything done for the week. I’m finding this is a bit too much information for a quick review and a manual assessment for the week where I aim to map out things for the upcoming week. It is a bit too much information overload. I don’t know what I’m going to do about it. Now that I have things automated and aggregated I need to rework my human side of things.

The Value of the Automation

The goal of my automations is to give me time back and remove the manual and grunt work so I can do the thinking work. I’ve done this with personal life and work life (and large systems and projects). The current round of automation and aggregation surfaced that I am doing more than I thought I was doing. Another goal is to give more time to do the human thinking and the projects and work needed. I do this so I can do the creative human work and the human thinking work (I find large errors in summarization in nearly all generative AI and creative work so I keep that at bey and use it for grunt work that I check). The scripted automations don’t need the hand holding and I have some occasional test run to check on them to see if they need attention and honing.

Automating the Daily Dump Note

In work and personal life I’ve long been an automator. It started in the ’90s, but actually started when I was a kid and talking with my dad about systems, making them more efficient, and removing steps, and automations. Any manual rote step I look at and think about making it a click or “just run”. I’ve used Obsidian since June 2020 and have a lot of things in my markdown notes trigged by TextExpander.

Reducing Steps Setting my My Daily Dump Note

Little by little I’ve been building out some automations for my note taking that sits under Obsidian’s watchful eye, but may start elsewhere or seeds start elsewhere. In the morning I run some automations that sweep up videos of interest I forwarded to myself from the prior evening or that morning that are in a personal instance of Slack. I have one that pulls in notes in Drafts with a specific tag then archives the note. One last piece I’d really like to add is to have my known items from across my calendars dropped in at the top of that Daily Dump note.

Getting the calendar dump is something I’d really like to have before I get to my laptop in the morning. But, each of these automations could and should be automated and dropped in the note before I fully get to it as well.

This takes one thing… having the note.

Creating the note with the series of headers and subheadings is something I usually do in a new markdown note with TextExpander each morning. But, I can’t automate with TextExpander (or I haven’t sorted out how to trigger that from a far).

Automating but Not with Apple Shortcuts

I started a couple days ago thinking that since I had Actions for Obsidian I could easily create a new Daily Dump with the correct future date as a title and have it trigger the Templater template for the note with the correct date. I quickly got the file created properly, but in over 4 hours that included a couple hours the following day it was still not working properly. This also included a few different apps and pieces and if any of them change it breaks.

Since I not only wanted to create one note with a template for the next day, but to have a few days ahead created (say a week at a time). I gave up on Shortcuts and in under 20 minutes I had a python script doing exactly what I wanted. This included setting up a cronjob to run it on Sunday evening and have a week of notes set.

Using the Output

This morning I opened Obsidian and one of the future day’s notes I had setup as tests was a click away and I was capturing an idea. This is a nice change and one I wished I had done long ago.

The next step is automating bringing in the aggregation automations notes and dropping that into my Daily Dump.

2025 Vanderwal.net Backend Modernization is Done

A couple years ago I thought I would update the backend code from PHP 5.6 to PHP 7 and initial progress on it was hindered by time available.

Planning the Modernization Work

A few weeks back I started looking at it again and mapped it out properly like a project. I realized PHP 7 was deprecated and I should really head to PHP 8, so that target was set. I was planning on keeping things relatively simple using a database connection quite similar to what I had used, but digging through PHP 8 books and resources on O’Reilly Learning Platform everything was using a newer more flexible method. After digging further I took the route that would take a bit more work modifying existing code (some going back to 2000 and 2001). But, as I dug into the work I realized I was only needing to modify and modernize about 20% to 30% of code on the pages and templates.

In doing this I also realized my old method of security around the system management backend was no longer working, so it had to be rewritten as well. That meant rebuilding the backend screens. Those updates went live two days ago on the 19th.

With that done it was back to the last third or so of the pages and templates that are public facing. I had already reworked the category output pages and adding pagination to them. No longer will all 121 Folksonomy categorized posts show up on one screen, only 15 at a time will. The “Personal” category has 369 posts (it is a blog so it is about me, you see, but just not all of it).

The RSS feed received a very minor update to RSS 0.92 to keep in line with many of the OG methods that remain.

The Actual Homepage has been Restructured

The homepage for vanderwal.net has been restructured to make it easier to find information that isn’t directly in the blog and I get emails and DMs about somewhat regularly. Moving it to two columns helped this. I do need to modify this to flex or grid CSS model as tweaking the layout was rather tedious.

This Modernization was like Changing the Plumbing and Wiring in a Building

This modernization was like bringing the plumbing and wiring of a building up to new building code. The walls and structure are all pretty much the same. The top layer stays the same for now.

This modernization does allow me to hopefully finish setting up webmentions, which I’ve had partly wired since around 2021 or so. I just need the last piece to that to work. There are also other IndieWeb related updates I’m planning on making and have been waiting to get this code updated before modifying and adding them into place. By the way, if you are running your own site and/or blog, the IndieWeb community has a gem. There are a lot of resources in their wiki and pages helping anybody with their own site.

The pagination for the blog is likely going to change from a date with month focussed pagination to a page model with the oldest selection being page 1. The archive page will get a long over due update so it doesn’t stop at 2003 (looks at calendar, yep it is out of date). I’m hoping to have an archive page that shows activity, but also addresses the different post types (essay, journal, and weblog) that only lasted the first few years, but also around the 2014 code update and site move the entry type template went missing.

The category listings pages will also likely get an update and the category page may likely get some ease of moving through the posts over time, beyond general pagination.

Assistance with the Update

This being 2025 the question pops up if and how I was using generative AI as part of this. I was using Claude.ai from Anthropic with some initial questions, then I’d head to O’Reilly’s resources to validate them and learn what I needed to know (it had been about 10 years since I was knee deep into PHP). When coding and modernizing the pages and templates I’d and hit defects I’d run those past Claude to sort out what the issue may be (sometimes missing “;”, others the new query wrapper and parsing method caused me to miss something, or I had deprecated code I hadn’t converted). Claude would point out my errors and instruct me how to correct it. Sometimes it would offer a few options for approaches (some were not quite right and others were good and I needed to select a path - after verifying and learning about them further). It also would crank out code. I gave Claude instructions not to bother with large chunks of my pages and code, which it left alone.

I use Claude stand alone and used is Project function to keep things focussed. I fed it the outlines and high level task areas I have in GitHub and Obsidian and it was keeping track of what was accomplished and how the work met the goals. The most impressive thing, compared to other generative AI options is it was very strict with identifying things not viable in PHP 8 (and its iterative versions) as nothing else did this well. Claude also had the code of pages and templates I had worked on and would point out I was using a structure and method in other page and ask if I shouldn’t use that practice on the page I just fed it to sort out some defect I was working through. My code has had four or more iterations over the 25 years and my early coding wasn’t so hot and still remained. Claude helped my code get more consistent, not by it fixing it, but pointing out I had something good and modern and I should keep consistent with that. By the last couple of templates I didn’t need to have Claude check them as they worked with my own editing, but I still fed them in as it seems to help improve suggestions and catching lack of consistency of my own doing.

A year ago I tried this with OpenAI and its ChatGPT and it was a hot mess. It couldn’t keep PHP versions correct. I try it with every update and I find it really problematic and what it outputs (code and other attempts) as nothing better than mediocre and often not correct.

IDE Use

In the last 10 to 15 years the IDE I’ve used to code and work on vanderwal.net has been from Panic and either Coda or now Nova, which have worked well. I have kept a good firewall between AI assistance and the IDE. I don’t mind type ahead suggestions. But, finding deprecated code to address was something I was going to need. Some friends suggested I try PhpStorm by JetBrains, which seemed good as I’ve used PyCharm a few times in the past and really enjoyed it. I knew I didn’t want VS Code near this, as I’ve pretty much had it with VS Code (I mostly use it with Python for data analytics) due to plug-in issues and lack of ease keeping projects separated.

I picked-up a trial of PHPStorm and after a day or so I had the hang of a good portion of what I needed to do. My favorite part is the setting the exact version of PHP you are working with. It highlights where there are errors and problems. In the last couple of days as I finally was getting the hang of PHP 8 and the methods I was regularly using PHPStorm was helping with type ahead suggestions (there were a few times where I accidentally triggered them when I didn’t want them and nearly turned of that functionality - control Z is your friend). PHPStorm also can make use of GitHub CoPilot, which I don’t find helpful with OpenAI connected to it, but is better with Claude Sonnet. The downside with CoPilot is it doesn’t have access to the Project space in Claude I’ve been working with and therefore its suggestions are less on target - CoPilot with Claude is light years better for PHP than OpenAI offerings). Essentially I didn’t use the incorporated genAI functionality and I was very happy with that setup.

Posting Ease

One of the things I’m looking forward to are slightly better methods for posting to this site and managing posts. Many of the steps beyond creating and posting are manual steps, like kicking off creation of the RSS feed (I do that after a quick review of the created post as it is live, I kick the RSS feed after that review). The alerting the media, or the alerts beyond basic RSS, is also a manual step done after that review. I may automate the combination of those two kicks after a review.

Weeknote - 18 October 2020

Okay, that week was the prior week’s weeknote. Now I’m trying to capture two weeks in one. The prior week was rather busy and the weekend full too.

The morning coffee walk, this week turned a bit wet and chilly. I may need to change from wearing shorts for my this trek to get me out my door and a bit of exercise to start the day. Seasons and other temporal changes of worldly transitions have really flown past this year with little acknowledgement. The trees are just starting to turn in their autumnal color pageant, but it seems like they were just bare and bright green sprouts coming out.

I got a note this week from my webhost, which had been bought quite a while ago by GoDaddy and they finally said they are transitioning and my host is going away. I know a lot of people who work at GoDaddy and the leadership and inhumane leadership problems are gone. But, they are planning on moving from a hosting plan and platform I love that fits what I want to keep going (this site) and some small experimental spaces playing with Python, NodeJS small services, and a little Ruby and moving to a service that really isn’t clear about what it does, nor what it offers, nor pricing, nor service, and it is only based in the UK. With Brexit it is deeply unclear what is going on in the UK with regulation and anything and that is one of the last places I would want to have anything hosted.

So, some of my time will be focussed in the next couple or few weeks transitioning elsewhere. I think I know where, which is a hosting platform from former founders and employees of my current host. They have similar offerings, but I’m needing to sort out what these changes will entail for some of the custom pieces I have and dealing with email.

I was in the midst of starting to plan an upgrade to the underlying code of the site to bring it to a modern version of PHP. This is on hold until I get the site moved.

Read

There wasn’t a lot of reading time this week. But, I sort of parked An Absolutely Remarkable Thing for now as the micro-fame discussions were something that was causing a lot of self reflection around similar. I picked up John Green’s The Fault in Our Stars and just a few pages in I’m happy with the swap as John Green’s writing voice is one I find comfort in.

I’m also reading / skimming back through some Richard Feynman as some friends have stumbled on to it and has lead to interesting discussions. I read Six Easy Pieces around 2003 or so after writing the draft of Model of Attraction and as I fleshed it out and it turned into Complexity / Social Lenses there is a strong underpinning in physics through Feynman’s introduction, followed by discussions with good depth in physics and quantum underpinnings.

Watched

The Pete Souza documentary, The Way I See It about his time as White House photographer for Reagan and Obama. It was completely wonderful and a solid reminder of what a great leader does through understanding things deeply and supporting all others through leading with empathy.

Listened

Tigran Hamasyan is a musician I stumbled upon through a “what is this” explainer on YouTube, which lead to a mini deep dive. The two videos that had been deeply intrigued and really enjoying his music are IMPOSSIBLE Time Signature or 4/4? Tigran Hamasyan Explained and The Rhythms of Tigran Hamasyan on David Bruce’s channel, which I have enjoyed and stumbled on before. The cross over and different mental model using math transformations and mapping patterns through size relevance patterns that are adaptive is really intriguing.

Food

I don’t understand why sole, particularly Dover or Petrale, is so hard to find on the East Coast. I swear they were pretty much a year round fish growing up on the West Coast. This week I stumbled on a decent sale on Dover Sole so made a quick fry in virgin olive oil and brown butter, with a dry coating of corn starch, rice crumbles, sea salt, and black pepper then finishing with lemon and quick fried capers and pickled capers. This was a good Sunday brunch to say the least.

Productivity

In this transition from light too mid-term notes in NValt to Obsidian for better organization and cross-linking and an app that actually works (NValt stopped working spectacularly). One of the things I was peeved about was the tagging I had done in NValt. But, Brett Terpstra knows tagging well and tucked the tags in the user interface of NValt into the tag field in Apple’s file metadata. The one that I’m really wanting to get organized is my blogfodder tag, which is really rough drafts of posts, or collections of notes no a subject.

Nokia to Nip Its Ecosystem?

First off, I admit it I like Nokia and their phones (it may be a bit more than like, actually). But, today's news regarding Nokia further refines development strategy to unify environments for Symbian and MeeGo is troubling, really troubling. Nokia is stating they are moving toward more of an app platform than software. It is a slight nuance in terms, but the app route is building light applications on a platform and not having access to underlying functionality, while software gets to dig deeper and put hooks deeper in the foundations to do what it needs. Simon Judge frames it well in his The End of Symbian for 3rd Party Development.

Killing A Valued Part of the Ecosystem

My love for Nokia is one part of great phone (voice quality is normally great, solidly built, etc.) and the other part is the software third party developers make. Nokia has had a wonderfully open platform for developers to make great software and do inventive things. Many of the cool new things iPhone developers did were done years prior for Nokia phones because it was open and hackable. For a while there was a python kit you could load to hack data and internal phone data, so to build service you wanted. This is nice and good, but my love runs deeper.

When my last Nokia (E61i) died after a few years, its replacement was a Nokia E72. I could have gone to iPhone (I find too many things that really bug me about iPhone to do that and it is still behind functionality I really like in the Nokia). But, the big thing that had me hooked on Nokia were two pieces of 3rd party software. An email application called ProfiMail and a Twitter client called Gravity. Both of these pieces of software are hands down my favorites on any mobile platform (BTW, I loathe the dumbed down Apple mail on iPhone/iPod Touch). But, I also get to use my favorite mobile browser Opera Mobile (in most cases I prefer Opera over Safari on iPhone platform as well). This platform and ecosystem, created the perfect fit for my needs.

Nearly every Nokia user I know (they are hard to find in the US, but most I know are in Europe) all have the same story. It is their favorite 3rd party applications that keep them coming back. Nearly everybody I know loves Gravity and hasn't found another Twitter client they would switch to on any other mobile platform. The Nokia offerings for email and browser are good, but the option to use that best meets your needs is brilliant and always has been, just as the unlocked phone choice rather than a carrier's mangled and crippled offering. If Nokia pulls my ability to choose, then I may choose a phone that doesn't.

Understanding Ecosystems is Important

Many people have trashed Nokia for not having a strong App Store like Apple does for iPhone. Every time I hear this I realize not only do people not understand the smartphone market that has existed for eight years or more prior to iPhone entering the market, but they do not grasp ecosystems. Apple did a smart thing with the App Store for iPhone and it solved a large problem, quality of applications and secondarily created a central place customers could find everything (this really no longer works well as the store doesn't work well at all with the scale it has reached).

While Apple's ecosystem works well, most other mobile platforms had a more distributed ecosystem, where 3rd party developers could build the applications and software, sell it directly from their site or put it in one or many of the mobile application/software stores, like Handango. This ecosystem is distributed hoards of people have been using it and the many applications offered up. When Nokia opened Ovi, which includes an app store with many offerings, many complained it didn't grow and have the mass of applications Apple did. Many applications that are popular for Nokia still are not in Ovi, because a prior ecosystem existed and still exists. That prior ecosystem is central what has made Nokia a solid option.

Most US mobile pundits only started paying attention to mobile when the iPhone arrived. The US has been very very late to the mobile game as a whole and equally good, if not better options for how things are done beyond Apple exist and have existed. I am really hoping this is not the end of one of those much better options (at least for me and many I know).

Understanding Folksonomy: Tagging that Works presentation posted

I have finally posted my Understanding Folksonomy: Tagging that Works (4MB PDF) presentation from d.construct. I have been presenting the content in this for nearly 2 years and have been iterating it. I have been wanting to get the Folksonomy Triad out in public as it has been getting really strong response in the 18 months that I have been using it (in the last 9 months I turned it into an animation that really drives the point home). I have not done a movie of the presentation yet, but that could be coming in the next few months.

I have not been posting the presentations, as the size has been increasing and when they get blogged elsewhere the bandwidth shoots up. But, I have been playing with Amazon S3 to store these files at lower cost and eat less bandwidth. I have been playing with the Python and Ruby scripts, but when Jeff Barr demonstrated the S3 Firefox Organizer I knew it was made really simple to upload, set permissions, and grab a sharable URL.

Developing the Web for Whom?

Google Web Developer Toolkit for the Closed Web

Andrew in his post "Reading user interface libraries" brings in elements of yesterday's discussion on The Battle to Build the Personal InfoCloud. Andrew brings up something in his post regarding Google and their Google Web Developer Toolkit (GWT. He points out it is in Java and most of the personal web (or new web) is built in PHP, Ruby [(including Ruby on Rails), Python, and even Perl].

When GWT was launched I was at XTech in Amsterdam and much of the response was confusion as to why it was in Java and not something more widely used. It seems that by choosing Java for developing GWT it is aiming at those behind the firewall. There is still much development on the Intranet done in Java (as well as .Net). This environment needs help integrating rich interaction into their applications. The odd part is many Intranets are also user-experience challenged as well, which is not one of Google's public fortés.

Two Tribes: Inter and Intra

This whole process made me come back to the two differing worlds of Internet and Intranet. On the Internet the web is built largely with Open Source tools for many of the big services (Yahoo, Google, EBay, etc.) and nearly all of the smaller services are Open Source (the cost for hosting is much much lower). The Open Source community is also iterating their solutions insanely fast to build frameworks (Ruby on Rails, etc.) to meet ease of development needs. These sites also build for all operating systems and aim to work in all modern browsers.

On the Intranet the solutions are many times more likely to be Java or .Net as their is "corporate" support for these tools and training is easy to find and there is a phone number to get help from. The development is often for a narrower set of operating systems and browsers, which can be relatively easy to define in a closed environment. The Google solution seems to work well for this environment, but it seems that early reaction to its release in the personal web it fell very flat.

13 Reasons

A posting about Top 13 reasons to CONSIDER the Microsoft platform for Web 2.0 development and its response, "Top 13 reasons NOT to consider the Microsoft platform for Web 2.0 development" [which is on a .Net created site] had me thinking about these institutional solutions (Java and .Net) in an openly developed personal web. The institutional solutions seem like they MUST embrace the open solutions or work seamlessly with them. Take any one of the technical solutions brought up in the Microsoft list (not including Ray Ozzie or Robert Scoble as technical solutions) and think about how it would fit into personal site development or a Web 2.0 developed site. I am not so sure that in the current state of the MS tools they could easily drop in with out converting to the whole suite. Would the Visual .Net include a Python, PHP, Ruby, Ruby On Rails, or Perl plug-in?The Atlas solution is one option in now hundreds of Ajax frameworks. To get use the tools must had more value (not more cost or effort) and embrace what is known (frogs are happy in warm water, but will not enter hot water). Does Atlas work on all browsers? Do I or any Internet facing website developer want to fail for some part of their audience that are using modern browsers?

The Web is Open

The web is about being browser agnostic and OS agnostic. The web makes the OS on the machine irrelevant. The web is about information, media, data, content, and digital objects. The tools that allow us to do things with these elements are increasingly open and web-based and/or personal machine-based.

Build Upon Open Data and Open Access

The web is moving to making the content elements (including the microconent elements) open for use beyond the site. Look at the Amazon Web Services (AWS) and the open APIs in the Yahoo Developer Network. Both of these companies openly ease community access and use of their content and services. This draws people into Amazon and Yahoo media and properties. What programming and scripting languages are required to use these services? Any that the developer wants.. That is right, unlike Google pushing Java to use their solution, Amazon and Yahoo get it, it is up to the developer to use what is best for them. What browsers do the Amazon and Yahoo solutions work in? All browsers.

I have been watching Microsoft Live since I went to Search Champs as they were making sounds that they got it too. The Live Clipboard [TechCrunch review] that Ray Ozzie gave at O'Reilly ETech is being developed in an open community (including Microsoft) for the whole of the web to use. This is being done for use in all browsers, on all operating systems, for all applications, etc. It is open. This seems to show some understanding of the web that Microsoft has not exhibited before. For Microsoft to become relevant, get in the open web game, and stay in the game they must embrace this approach. I am never sure that Google gets this and there are times where I am not sure Yahoo fully gets it either (a "media company" that does not support Mac, which the Mac is comprised of a heavily media-centric community and use and consume media at a much higher rate than the supported community and the Mac community is where many of the trend setters are in the blogging community - just take a look around at SXSW Interactive or most any other web conference these days (even XTech had one third of the users on Mac).

Still an Open Playing Field

There is an open playing field for the company that truly gets it and focusses on the person and their needs. This playing field is behind firewalls on Intranet and out in the open Internet. It is increasingly all one space and it continues to be increasingly open.

More XTech 2006

I have had a little time to sit back and think about XTech I am quite impressed with the conference. The caliber of presenter and the quality of their presentations was some of the best of any I have been to in a while. The presentations got beneath the surface level of the subjects and provided insight that I had not run across elsewhere.

The conference focus on browser, open data (XML), and high level presentations was a great mix. There was much cross-over in the presentations and once I got the hang that this was not a conference of stuff I already knew (or presented at a level that is more introductory), but things I wanted to dig deeper into. I began to realize late into the conference (or after in many cases) that the people presenting were people whose writting and contributions I had followed regularly when I was doing deep development (not managing web development) of web applications. I changed my focus last Fall to get back to developing innovative applications, working on projects that are built around open data, and that filled some of the many gaps in the Personal InfoCloud (I also left to write, but that did get side tracked).

As I mentioned before, XTech had the right amount of geek mindset in the presentations. The one that really brought this to the forefront of my mind was on XForms, an Alternative to Ajax by Erik Bruchez. It focussed on using XForms as a means to interact with structured data with Ajax.

Once it dawned on me that this conference was rather killer and I sould be paying attention to the content and not just those in the floating island of friends the event was nearly two-thirds the way through. This huge mistake on my part was the busy nature of things that lead up to XTech, as well as not getting there a day or two earlier to adjust to the time, and attend the pre-conference sessions and tutorials on Ajax.

I was thrilled ot see the Platial presentation and meet the makers of the service. When I went to attend Simon Willison's presentation rather than attending the GeoRSS session, I realized there was much good content at XTech and it is now one on my must attend list.

As the conference was progressing I was thinking of all of the people that would have really benefitted and enjoyed XTech as well. A conference about open data and systems to build applications with that meet real people's needs is essential for most developers working out on the live web these days.

If XTech sounded good this year in Amsterdam, you may want to note that it will be in Paris next year.

New Hosting and Peace of Mind

Most of the e-mail coming in is now coming through the new hosting company. I am getting feedback from people that they can see this new site. The change of the domain to the new hosting started resolving correctly in 30 minutes for me, which was very fast.

The choice in hosting came down to two strongly recommended companies, Pair.com and Segment Publishing (SegPub). Both had nothing, but positive recommendations and each had more than four recommendation (Pair actually had over 10 recommendations). Both offered the services I wanted and needed (PHP, Python, secure mail, MySQL, solid up-time records, and Rails (Rails was more of a nice than a need)).

I chose SebPub based on two things that separated it from Pair. One was price, SegPub was a little less and offered a good introduction and trial period. It is an Australian company (servers in the US) and payments are in Australian dollars. Secondly, the first person to recommend SebPub put me in contact with the guy who runs SegPub in a chat. I was able to asks a lot of questions specific to my needs and he was able to easily point me to the needed information or just provide it in the chat. That personal touch was an incredible sell. Just knowing there is that access helped. I had really grown tired of waiting a few hours between responses to trouble tickets and e-mail exchanges and not getting resolution and just getting the ticket closed.

Getting things prepared and moved over took the usual amount of time, but I had far fewer bumps than I thought I would have had. I needed to better optimize MySQL to perform better in MySQL4. I was to the point I needed to make the hosting change and the time disruption was not welcome, but I knew I could focus when the transition was over and done.

So far I have been incredibly happy. I also know that Pair would also be a very good company to go to as well. This option gives me great peace of mind.