Abstract

This is an effort to describe the current scenario of the FreeBSD Documentation set, discuss some ideas and designs, then make a plan for future work. It’s written after the transition to Hugo/Asciidoctor.

1. Current Status

1.1. Toolkit

1.1.1. Hugo

Hugo is a static HTML and CSS website generator written in Go. It is optimized for speed, ease of use, and configurability. Hugo takes a directory with content and templates and renders them into a full HTML website.

Hugo relies primarily on Markdown files with front matter for metadata. It also supports other content formats like Asciidoctor, which is what FreeBSD uses because it has more flexibility for writing complex texts.

Hugo is designed to work well for any kind of website including blogs, tumbles, and docs.

1.1.2. Asciidoctor

Asciidoctor is a fast, open source, Ruby-based text processor for parsing AsciiDoc into a document model and converting it to output formats such as HTML 5, DocBook 5, manual pages, PDF, and EPUB 3.

1.1.3. Source Code

Source code hosted on FreeBSD git repository doc.

It has two main Hugo projects, documentation and website, each one on its respective directory, with a shared directory in between. Most Hugo contents files are in Asciidoctor format, and that are a few .toml files in the repository as well.

1.1.4. Scripts

Currently, both projects need a few helper scripts written in Python, Ruby, and sh. These scripts are hosted in tools directories.

1.2. Documentation

Set of Articles and Books created by the project and community. All documents are in Asciidoctor format.

Articles have a single .adoc file.

Books are a mix of single .adoc and multiple .adoc files.

Following are more information about the current scenario of the Documentation:

1.2.1. Features

Online Documentation
Offline Documentation
  • HTML

    There is a make html target in Makefile.

    Assets on ftp/doc are outdated.

  • PDF

    There is a make pdf target in Makefile.

    Assets on ftp/doc are outdated.

  • EPUB and Other formats

    There is a make epub target in Makefile (Experimental).

Ports/Packages

Doceng Ports misc/freebsd-doc-*.

Releases

Not sure how it is or how it will be the process.

Translations

Mix of .adoc (directly) and Online translations through Weblate.

Work in progress, there are open issues.

1.2.2. Points of Attention

Hard-coded language on Includes (Documentation)

|Closed|

Describe the issue:

This is an example of Includes on an article, which contains the en part fixed:

 include::shared/authors.adoc[]
 include::shared/en/teams.adoc[]
 include::shared/en/mailing-lists.adoc[]
 include::shared/en/urls.adoc[]
  • Update 2021-10-05:

    The review D31926, Build offline documentation using Hugo handle this and other points of attention.

  • Update 2021-11-06:

    Review D31926 committed in 64acd16.

Building HTML with Hugo (Online Docs) and Asciidoctor Standalone (Offline Docs)

|Closed|

Describe the issue:

  1. Not possible to difference an HTML build from Hugo and Asciidoctor standalone.

      // This is for Hugo:
     ifeval::["{backend}" == "html5"]
      include::shared/authors.adoc[]
      include::shared/{{% lang %}}/teams.adoc[]
      include::shared/{{% lang %}}/mailing-lists.adoc[]
      include::shared/{{% lang %}}/urls.adoc[]
     endif::[]
    
    
      // This is for Asciidoctor standalone:
      ifeval::["{backend}" == "html5"]
      include::shared/authors.adoc[]
      include::shared/{lang}/teams.adoc[]
      include::shared/{lang}/mailing-lists.adoc[]
      include::shared/{lang}/urls.adoc[]
     endif::[]
  2. HTML output are different

    • Update 2021-09-13:

      There is a Review/WIP (D31926) for building offline documentation using Hugo; it covers Documentation and Website.

    • Update 2021-10-05:

      Review D31926 has been updated; it seems a good path to follow.

    • Update 2021-11-06:

      Review D31926 committed in 64acd16.

|Open issue| |Needs Test/PoC|

Describe the issue:

This can be seen when building the documentation with:

$ make documentation
[...]
$ ls documentation/public/

index.html:

<html>
  <head>
    <title>https://docs.freebsd.org/en/</title>
    <link rel="canonical" href="https://docs.freebsd.org/en/"/>
    <meta name="robots" content="noindex">
    <meta charset="utf-8" />
    <meta http-equiv="refresh" content="0; url=https://docs.freebsd.org/en/" />
  </head>
</html>

The Documentation index redirects all requests to "https://docs.freebsd.org/en/".

And we have other places with hard-coded links, like css, images, js, and to other documents:

$ cd public/en

$ grep -nir docs.freebsd.org *
articles/problem-reports/index.html:9:    <link rel="canonical" href="https://docs.freebsd.org/en/articles/problem-reports/" />
articles/problem-reports/index.html:14:    <link rel="shortcut icon" href="https://docs.freebsd.org/favicon.ico">
articles/problem-reports/index.html:19:    <link rel="stylesheet" href="https://docs.freebsd.org/css/docbook.css">
articles/problem-reports/index.html:20:    <link rel="stylesheet" href="https://docs.freebsd.org/css/font-awesome-min.css">
articles/problem-reports/index.html:24:    <meta name="twitter:domain" content="docs.FreeBSD.org"/>
articles/problem-reports/index.html:30:    <meta property="og:image" content="https://docs.freebsd.org/favicon.ico"/>
articles/problem-reports/index.html:33:    <meta property="og:url" content="https://docs.freebsd.org/en/articles/problem-reports/" />
articles/problem-reports/index.html:39:        "url": "https:\/\/docs.freebsd.org\/en\/articles\/problem-reports\/",
articles/problem-reports/index.html:47:    <script src="https://docs.freebsd.org/js/google.js"></script>
articles/problem-reports/index.html:251:<p>In either case, following the process described in <a href="https://docs.freebsd.org/en/books/porters-handbook/#port-upgrading">Porter’s Handbook</a> will yield the best results. (You might also wish to read <a href="https://docs.freebsd.org/en/articles/contributing/#ports-contributing">Contributing to the FreeBSD Ports Collection</a>.)</p>
articles/problem-reports/index.html:282:<p>If the problem is in the base system, first read the FAQ section on <a href="https://docs.freebsd.org/en/books/faq/#LATEST-VERSION">FreeBSD versions</a>, if you are not already familiar with the topic.
[...]

This is possible the same issue we are having on the website mirrors.

Possible solutions:

One idea would be to have different Hugo configurations (Needs to investigate it better).

  • Update 2021-09-26:

    Review/WIP (D31926) for building offline documentation using Hugo also covers this.

  • Update 2021-11-06:

    Review D31926 committed in 64acd16.

Offline Documentation Outdated (FTP/DOC)

|Open issue|

Describe the issue:

On download.freebsd.org/ftp/doc is released the whole set of Articles and Books of the FreeBSD Documentation in several formats and for all languages:

  • article.epub

  • html-split.tar.bz2

  • article.html-split.tar.zip

  • article.html.tar.bz2

  • article.html.tar.zip

  • article.pdf

  • article.pdf.bz2

  • article.pdf.zip

  • article.txt.bz2

  • article.txt.zip

These files are not updated since the migration to Hugo/Asciidoctor.

We don’t need all formats as we had, but at least HTML and PDF are necessary.

Possible solutions:

Close the attention point [Building HTML with Hugo and Asciidoctor] and work with the Cluster Administrators to create a new script to build it.

In practice, we will create a new cron script in the build jail and put the output in an additional directory.

  • Update 2021-11-06:

    Add option to archive/compress Documentation/HTML offline files (038be2aa)

1.3. Website

The FreeBSD Website hosts several types of information, and many teams need to update it.

Following are more information about the current scenario of the Website:

1.3.1. Features

Online Website
Mirrors

The Official Mirrors page has several mirrors not working; the exceptions are not usable because of issues with hard-coded links/redirects. More info about mirrors on Mirroring FreeBSD article.

The Online Website currently builds with Hugo. It’s the same content that is sent to website mirrors.

Translations

Currently, only direct translation through source code is supported.

1.3.2. Points of Attention

Website mirrors not working

|Open issue| |Needs a Definition|

Describe the issue:

Website mirrors are not usable at this moment, probably because of the point of attention hard-coded links, described in the Documentation section.

Possible solutions:

The same on hard-coded links.

  • Update 2021-09-26:

    Stopping offering www mirrors for the FreeBSD website. This is an old feature to save bandwidth and to increase speed access for users. We can talk with Cluster Administrators and include the FreeBSD website (www.freebsd.org and docs.freebsd.org) on GeoDNS infrastructure.

1.4. General

1.4.1. Points of Attention

Fewer people familiar with the new toolchain

|Open issue|

Describe the issue:

Possible many committers and also doceng members are not fully familiar with Hugo/Asciidoctor.

Possible solutions:

Online workshops and/or videos about.

Sergio is willing to do this with help of others.

  • Brief introduction to the Documentation;

  • Take a bug from Bugzilla and show how to fix it;

  • Make a slight change in the Documentation;

  • Make a change to the Theme;

2. Future

2.1. Documentation

2.1.1. Documentation Portal

Redesign of the Documentation Portal.

Create a new design, responsive and with global search. Work in progress by Sergio.

2.2. Website

2.2.1. Manual Pages

Scripts to generate the HTML pages using mandoc.

2.2.2. Ports page on Web

Ports scripts to create an applications portal.

2.2.3. Redesign of the Website

New design, responsive and dark theme.

2.3. General

2.3.1. Edit this Page button

A button Edit this Page can help in getting more people involved. With one click, it leads the reader to the source code.

Here are some examples of documentation portals that use this feature:

2.3.2. Encourage GitHub pull requests

This is already supported, and all steps necessary to accept/land a pull request are described in the Committers Guide.

  • Create a template for submitting pull requests.

  • Encourage doc committers to work with GitHub pull requests.

3. Next Steps

  1. Finish this document

    Responsible: Everyone

    Deadline: September 20, 2021 ?

  2. Second item

    Responsible: ?

    Deadline: ?