Auto-updating vendor information

I track information about companies. For myself, for others that pay me, for fun. And I’m pretty good at it. But it still requires I spend time I wouldn’t need to if I had some kind of auto-updating vendor information system!

What do I mean by that? Well, vendor sheets rarely change, in my experience. Whenever I enter into a commercial transaction with a company, it’s information rarely changes. That’s great for me, since I keep text files on the vendors and copy and paste them into the docs of whichever project I’m working on.

Well, I’ve imagined something kinda neat, and we’ll be able to apply the lessons to more fun stuff that vendor data, but here we go, embracing some web tech and data communities, we’re gonna build our useful tool!

I even have part of it ready for demo, at Vendors.

Those are my initial vendors, with rather sparse data included.

The next part will be me implementing my site deployment method so my templates are available online, but here is the template that produces each of those articles in the list:

<article>
    {{ $wikidataid := .Params.wikidataid -}}
    {{- $wikidata := getJSON "https://www.wikidata.org/w/api.php?action=wbgetentities&ids=" $wikidataid "&languages=en&format=json" -}}
    {{- $data := index $wikidata "entities" $wikidataid -}}
    {{- $claim := index $wikidata "entities" $wikidataid "claims" -}}
    
    <h2>{{ .Title }} {{ with index $data "descriptions" "en" "value" }}({{ . }}){{- end -}}</h2>
    {{ with $claim.P856 -}}<p>Website: {{ index $claim "P856" 0 "mainsnak" "datavalue" "value" | markdownify }}</p>{{- end }}
    {{ with $claim.P1581 -}}<p>Blog: {{ index $claim "P1581" 0 "mainsnak" "datavalue" "value" | markdownify }}</p>{{- end }}
    {{ with $claim.P9138 -}}<p>Status website: {{ index $claim "P9138" 0 "mainsnak" "datavalue" "value" | markdownify }}</p>{{- end }}
</article>

Here’s what has happened so far:

  1. I make a list of vendors used on a single project
  2. I found all the vendors on wikidata.org
  3. Of the available data, I found some claims (website, blog, status website), and conditionally list them if present

My next steps are to learn the practices on Wikidata to add data. For instance, URLs are considered self-evident references, but I want to add other data to each vendor as well, which will require knowing how each claim is verified. Also, there are WikiProjects (Wikidata:WikiProjects - Wikidata), teams working on Wikidata, and there is Wikidata:WikiProject Companies - Wikidata, where I’ll start asking for mentoring in adding data.

The immediate next step is to add the three URLs (if available), to each of their Wikidata entries. When I’ve done that, those records will be updated on a subsequent site build (which I’ll probably do daily, but cache results for much longer, as I don’t expect Wikidata to change much after I’ve added a vendor and updated it, myself.

Anyhow, I hope you see the beginning of what I’m doing here. It’s pretty exciting. :slight_smile:

Had trouble with mod_security and fossil working together, but I got it figured out and now you can see the code for the vendors section on the site: maiki.xyz website: Check-in [10b90671de]

You can see my vendor files are just a title and a Wikidata ID (maiki.xyz website: mailgun.md at [10b90671de]):

---
title: "Mailgun"
wikidataid: "Q105539224"
---

Now that I can show my work, I’ll move forward with the next step. :slight_smile:

My next task is to create single templates for vendors, that is: a template to show the single page with vendor information. It will likely render at something like maiki.xyz/vendors/mailgun.

However, as I sit down I realize I need to have a better grasp of which Wikidata properties I’ll be including. I can front load a good portion of what I desire/what’s available by doing the research now, rather than constantly editing the templates; I’m not sure how I’m going to generate the custom outputs for these items, but I can see a lot of duplication, hence figuring out properties now.

For research I intend to read up on the suggested properties (I’m pretty familiar with them at this point), but also to seek out the most filled out companies on Wikidata, so I may see what’s possible. And of course I will be enumerating which properties I think might be useful, and then ask for an equivalent or propose new ones.

Okay, I’ve looked at the most complete data profiles for companies on Wikidata and have a set of properties I will use to fill in my vendors. Here they are:

There is one property I can’t find, something like “customer portal”: the URL to the customer dashboard for a given company. After I’ve filled out as much as possible from the above properties I’ll look at discussing and adding the property, or finding an existing one for that purpose. :slight_smile:


I read all the discussion pages for those properties, and found a few others based on recommendations from them:

I’ve used the properties I collected and added information to the entry for Mailgun. I rebuilt the site, and now you can see the difference, from before…

…to now:

:grimacing:

It’s cool, though! I added that blog URL to wikidata.org! And now it is available to everyone. :slight_smile:


Also, two properties to find:

  • support community
  • wiki

Okay, so I keep a minimal style on my website. I plan to add a bit more, but it will basically be the same. However, there is a lot of interesting things one can do with minimal HTML… such as using details and summary to create an easier to navigate directory of items (Vendors):

Another screenshot of an item “open”:

It was a surprisingly small change to make: maiki.xyz website: Check-in [c417b07588]

In other news, I’ve started moving the properties into partial templates, so they are easier to code and make editing the vendors template easier… though I still haven’t decided exactly where to stash code in this regard.

:thinking:

I’m having difficulty figuring out how to layout the data, as I don’t know what I will have generally available, from the list of properties I’ve found. So instead I’m going to go through and add a place for each property and then see what I’m dealing with. :slight_smile:

Okay, added several properties (maiki.xyz website: Check-in [b6ab36ef2b]), now I see how “complete” the current data is:

As far as public data goes, those are the properties I’ve found that may be useful for vendor contact sheets. The idea being I can drop a link in from my site, and describe how the project uses the vendor; docs are then done, and I don’t need to track down any additional info.

So with what I have I can lay out in a kind of card format… okay, it’s starting to make sense.

I need a property for “official wiki” or “knowledge base”.

I’ve created the first phase of oembed support for vendor docs maiki.xyz website: Check-in [2c2f35efd3]

It is very slow going, as I figure out how to configure hugo to output what I need. But making progress. :slight_smile:

I’m generating oEmbed auto-discovery on vendors, but I still need to template the single views for each one, so it makes sense to have a link back. That will be my next task.

Let’s see if this does anything interesting…

Hmmm… that isn’t exactly what I want… okay more hacking on the oembed implementation!