Archiving a really stubborn icon archive.

[GlitchyZorua]

Hi! so. I found this really old icon browser on the old internet, which needs to be archived.

It won't be easy though, because... how it was programmmed to get an icon, is very.. very 90s

You can visit this icon browser here: https://www.ibiblio.org/gio/iconbrowser/



You see, how this icon browser worked is, that, if you wanted to get an icon, you click on a table, then you click on the icon that you want. Simple, right?

Fucking wrong.



You see, the entire table shown here is all in one image. and to get an icon that you want, the website has to KNOW where your cursor is at.

like the POSITION of where it is.

Don't believe me? Here is a video

http://www.ibiblio.org/iconbin/imagemap/icon2?x,y <-- The x and y, shown here, is the coordinates of your cursor. each pixel of the image is a link to... an icon. If you tried to click on something else in the image that isn't an icon, it just throws an server-side error.




That's right. This icon browser does things SERVER SIDE.

To make things worse, you couldn't even search for icons, because that function was broken for whatever reason.




In conclusion, this is the worst icon repository in i've ever seen. and before you say that, "well, better html tags didn't exist back then", i'd like to remind you that sites like this one exist. All of these things that this site does effectively makes it a pain in the ass to archive.

So if you all have the energy to do it, please help me figure out a way to archive this icon archive.


Of course, when this entire library of icons gets archived by someone, or whatever, i'll be sure to tell melon to add it all over to Pixelsea.

[✿ Passerine ✿]

Archiving thousands of tiny icons, this sounds like the perfect Sisyphean task to occupy my time.

I'm not really familiar with coding at all, I have very surface level technical understanding, but would it be possible to code a bot to automate the archival process, like having it ping each icon URL and download the icon to a drive?

Again, I might just be speaking outta my rear-end here, but worth a shot. 

[✿ Passerine ✿]

Ok, had a bit of a brain blast

You know those programs that let you have a bot that can click specific parts of a webpage to interact with buttons automatically when you give them a command? I'm thinking that a similar bot could be applicable here, one that would click on a pre-set list of coordinates that correspond with each icon. Each page that has icons on it is organized in a standardized way, so a new set of coordinates would not need to be made for each page. After the bot clicks on an icon, it records the URL for the icon it clicked on, then goes back to the previous page and clicks on the next icon. It repeats this process until it runs out of coordinates/icons to click on.

After every single individual-icon URL is recorded, have another bot ping all of those URLS and download the icon images stored on them to a drive. It would take quite some time, but it would be faster and less labor intensive than doing it manually. There's an estimated 7,296 icons that need archiving.

There is also just the option of getting a team of people together to manually archive and organize every icon, which would be a colossal pain in the rear, but it is POSSIBLE.

[✿ Passerine ✿]

Ok, on mobile now. Had a realization after reading your post more carefully. I realize that the link itself contains the coordinates that then lead to a specific icon. Correct me if I’m wrong.

 Instead of needing to say, have a bot manually click on the page to load the URL for the icon, just make a coordinate list for all of the icons on all of the icon grid pages using a program, thus generating thousands of icon URLs. Then have a bot go through each link and save the gif file.

[Dan Q]

Gottit. Server-side image maps.

But the pattern is simple enough. The URLS are all in this format:

https://www.ibiblio.org/iconbin/imagemap/icon{i}?{y},{x}

Where {i} is a number from {1, 2, 3, 4, 5, 6}, and {y} and {x} are the coordinates on the image maps. The grids are all laid out the same, so we can sample them. Each grid has 8 images across (x dimension) by 8 down (y dimension). So we can take the width and height of those image grids (576 by 712) and divide then by 8 to get the size of the hit areas (72 by 89), choose a midpoint within those (36 by 44), and then add the width/height to permute a set of "hit targets". I get x = {36, 108, 180, 252, 324, 396, 468, 540}; y = {44, 133, 222, 311, 400, 489, 578, 667}. To test my logic I hit the "top left" and "bottom right" of those and they worked.

Now it's time to generate the full list. I used the following Ruby code to print a complete list of URLs.

Code

puts %w{1 2 3 4 5 6}.flat_map { |i|
  %w{36 108 180 252 324 396 468 540}.map { |x|
    %w{44 133 222 311 400 489 578 667}.map { |y|
      "https://www.ibiblio.org/iconbin/imagemap/icon#{i}?#{y},#{x}"
    }
  }
}.join("\n")

That worked nicely. There's a few URLs at the very end that resolve to "none", but we'll just ignore them later. Anyway: each page you hit at those URLs is actually a redirect. We could get our downloader to follow them, but I figured I might as well just get the actual URLs. I adapted the Ruby program:

Code

#!/bin/env ruby
require 'uri'
require 'net/http'

urls = %w{1 2 3 4 5 6}.flat_map { |i|
  %w{36 108 180 252 324 396 468 540}.flat_map { |x|
    %w{44 133 222 311 400 489 578 667}.flat_map { |y|
      "https://www.ibiblio.org/iconbin/imagemap/icon#{i}?#{y},#{x}"
    }
  }
}

# For each URL, make a HTTP request and expect a HTTP 302 response with a Location: pointing to the
# REAL URL. Remap the URLs:
urls.map! do |url|
  uri = URI(url)
  res = Net::HTTP.get_response(uri)
  if res.code == '302'
    res['location']
  else
    nil # Not sure what to do for the best if this happens! Let's ignore for now.
  end
end

# Some URLs end /none.html. They're invalid. Drop them.
urls.reject! { |url| url.nil? || url.end_with?('/none.html') }

# Output all the URLs
puts urls.join("\n")

That's gonna be a bit slow as it crawls through 8 x 8 x 6 = 384 different URLs to work out where each redirects to, so I'll post this thread as-is for now and come back to it when it's finished...

[Dan Q]

Ooh, I just realised that there aren't 6 pages of icons, there are 114! So I've adjusted my program; also fixed a couple of minor bugs. Now it does the file write by itself, freeing up STDOUT to show its progress:

Code

#!/bin/env ruby
require 'uri'
require 'net/http'

File.open('results.txt', 'w') do |f|
  urls = (1..114).flat_map { |i|
    %w{36 108 180 252 324 396 468 540}.flat_map { |x|
      %w{44 133 222 311 400 489 578 667}.flat_map { |y|
        "https://www.ibiblio.org/iconbin/imagemap/icon#{i}?#{x},#{y}"
      }
    }
  }

  # For each URL, make a HTTP request and expect a HTTP 302 response with a Location: pointing to the
  # REAL URL. Write these to our file.
  urls.each do |url|
    print url
    uri = URI(url)
    res = Net::HTTP.get_response(uri)
    if res.code == '302'
      if res['location'].end_with?('/none.html')
        puts " -> ⚠️ FAILED! Redirects to #{res['location']}"
      else
        puts " -> #{res['location']}"
        f.puts res['location']
      end
    else
      puts " -> ⚠️ FAILED! HTTP response code #{res.code}"
      nil # Not sure what to do for the best if this happens! Let's ignore for now.
    end
  end
end

I'll get that kicked-off and get back to you...

[Dan Q]

Generating this final URL list is going to be a while... I got up, had a shower, ten went and read a blog post or two and came back to find that it was only up to page 26 (out of 114) of icons. And I've still got to do the actual downloading. There's a lot! But it's getting there.

Meanwhile, I came up with another approach: you could just download the "grid" images and then use Imagemagick to cut them up! There's a risk that you get lower-quality copies, though, especially if any of the icons use lots of colors. Anyway; let's put that one on the backburner in case it's needed.

I also tried to trick the webserver into exposing the contents of its mapping files, but no dice.

But we've got something that will work, if slowly. More to follow, I'm sure...

[Dan Q]

Okay! I've got a full list of image URLs; attached. I'm gonna do something with them presently...

[Dan Q]

Got all the URLs. Now I've thrown together a (resumable) bulk-downloader script that uses wget to pull all of the icons, keeping them in a directory structure sorta-reminiscient of their original one:

Code

#!/bin/env ruby
require 'fileutils'

urls = File.read('results.txt').split("\n")
urls.each do |url|
  if url =~ /(\d+)\/([^\/]+?)$/
    dir, filename = $1, $2
    FileUtils.mkdir_p("icons/#{dir}")
    output = "icons/#{dir}/#{filename}"
    puts "#{url} -> #{output}"
    if File.exist?(output)
      puts "👍 File already exists, skipping download"
    else
      `wget --quiet --output-document=#{output} #{url}`
    end
  else
    puts "⚠️ Could not parse URL #{url}"
  end
end

My thinking is that I can create a Git repo, dump them all into there (so they're easy for anybody to get in bulk), and then configure a static site generator to present them in a similar way to the original (but without the server-side image maps, and perhaps with a client-side Lunr-powered search to replace the one that was broken.

More to follow, of course...

[Dan Q]

Okay! Here's a replica I've built of the icon library: ibiblio-icon-archive.danq.dev.

It can be browsed just like the original, except it uses the actual images rather than an imagemap'd collage of them, so the links are less-mysterious. Also I fixed search: type part of a filename and you'll find icons that contain it: basic, but functional.

The entire site, plus all the code I used to make it (most of it described above) it in a public Git repository, so you can easily download the whole thing if what you wanted was just to have all the icons. Some of them have come out all-black (because they were in the original), and one of them had an invalid character in its name (which prevented it from being downloaded from the original), but otherwise this archive is completely intact.

[Dan Q]

And now, of course, I've blogged about everything we achieved here. Nice nice nice.

Can't imagine it'll drive the Internet wild as much as yesterday's blog post which is still making my phone ping several times an hour as people comment on it or email be about it! But honestly, I think rescuing 7,200 GIFs is a big deal so I'm glad we've done it!

[✿ Passerine ✿]

I’m very grateful for the work you’ve done here  , hopefully OP will log back on soon and these icons can be added to the sea of pixels. I’ll take a personal copy of the icons to add to my own archives.