actions
/
mdBook
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Rewrote a large proportion of the Preprocessor docs to be up-to-date

This commit is contained in:
Michael Bryan 2018-09-25 19:41:38 +08:00
parent adec78e7f5
commit b1c7c54108
No known key found for this signature in database
GPG Key ID: E9C602B0D9A998DC
5 changed files with 244 additions and 147 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

104
book-example/src/for_developers/preprocessors.md
View File

@ -11,68 +11,71 @@ the book. Possible use cases are:
mathjax equivalents
## Implementing a Preprocessor
## Hooking Into MDBook
A preprocessor is represented by the `Preprocessor` trait.
MDBook uses a fairly simple mechanism for discovering third party plugins.
A new table is added to `book.toml` (e.g. `preprocessor.foo` for the `foo`
preprocessor) and then `mdbook` will try to invoke the `mdbook-foo` program as
part of the build process.
```rust
pub trait Preprocessor {
fn name(&self) -> &str;
fn run(&self, ctx: &PreprocessorContext, book: Book) -> Result<Book>;
fn supports_renderer(&self, _renderer: &str) -> bool {
true
}
}
While preprocessors can be hard-coded to specify which backend it should be run
for (e.g. it doesn't make sense for MathJax to be used for non-HTML renderers)
with the `preprocessor.foo.renderer` key.
```toml
[book]
title = "My Book"
authors = ["Michael-F-Bryan"]
[preprocessor.foo]
# The command can also be specified manually
command = "python3 /path/to/foo.py"
# Only run the `foo` preprocessor for the HTML and EPUB renderer
renderer = ["html", "epub"]
```
Where the `PreprocessorContext` is defined as
In typical unix style, all inputs to the plugin will be written to `stdin` as
JSON and `mdbook` will read from `stdout` if it is expecting output.
The easiest way to get started is by creating your own implementation of the
`Preprocessor` trait (e.g. in `lib.rs`) and then creating a shell binary which
translates inputs to the correct `Preprocessor` method. For convenience, there
is [an example no-op preprocessor] in the `examples/` directory which can easily
be adapted for other preprocessors.
<details>
<summary>Example no-op preprocessor</summary>
```rust
pub struct PreprocessorContext {
pub root: PathBuf,
pub config: Config,
/// The `Renderer` this preprocessor is being used with.
pub renderer: String,
}
// nop-preprocessors.rs
{{#include ../../../examples/nop-preprocessor.rs}}
```
</details>
The `renderer` value allows you react accordingly, for example, PDF or HTML.
## Hints For Implementing A Preprocessor
## A complete Example
By pulling in `mdbook` as a library, preprocessors can have access to the
existing infrastructure for dealing with books.
The magic happens within the `run(...)` method of the
[`Preprocessor`][preprocessor-docs] trait implementation.
For example, a custom preprocessor could use the
[`CmdPreprocessor::parse_input()`] function to deserialize the JSON written to
`stdin`. Then each chapter of the `Book` can be mutated in-place via
[`Book::for_each_mut()`], and then written to `stdout` with the `serde_json`
crate.
As direct access to the chapters is not possible, you will probably end up
iterating them using `for_each_mut(...)`:
Chapters can be accessed either directly (by recursively iterating over
chapters) or via the `Book::for_each_mut()` convenience method.
```rust
book.for_each_mut(|item: &mut BookItem| {
if let BookItem::Chapter(ref mut chapter) = *item {
eprintln!("{}: processing chapter '{}'", self.name(), chapter.name);
res = Some(
match Deemphasize::remove_emphasis(&mut num_removed_items, chapter) {
Ok(md) => {
chapter.content = md;
Ok(())
}
Err(err) => Err(err),
},
);
}
});
```
The `chapter.content` is just a string which happens to be markdown. While it's
entirely possible to use regular expressions or do a manual find & replace,
you'll probably want to process the input into something more computer-friendly.
The [`pulldown-cmark`][pc] crate implements a production-quality event-based
Markdown parser, with the [`pulldown-cmark-to-cmark`][pctc] allowing you to
translate events back into markdown text.
The `chapter.content` is just a markdown formatted string, and you will have to
process it in some way. Even though it's entirely possible to implement some
sort of manual find & replace operation, if that feels too unsafe you can use
[`pulldown-cmark`][pc] to parse the string into events and work on them instead.
Finally you can use [`pulldown-cmark-to-cmark`][pctc] to transform these events
back to a string.
The following code block shows how to remove all emphasis from markdown, and do
so safely.
The following code block shows how to remove all emphasis from markdown,
without accidentally breaking the document.
```rust
fn remove_emphasis(
@ -107,3 +110,6 @@ For everything else, have a look [at the complete example][example].
[pc]: https://crates.io/crates/pulldown-cmark
[pctc]: https://crates.io/crates/pulldown-cmark-to-cmark
[example]: https://github.com/rust-lang-nursery/mdBook/blob/master/examples/de-emphasize.rs
[an example no-op preprocessor]: https://github.com/rust-lang-nursery/mdBook/blob/master/examples/nop-preprocessor.rs
[`CmdPreprocessor::parse_input()`]: https://docs.rs/mdbook/latest/mdbook/preprocess/trait.Preprocessor.html#method.parse_input
[`Book::for_each_mut()`]: https://docs.rs/mdbook/latest/mdbook/book/struct.Book.html#method.for_each_mut

24
examples/de-emphasize.rs
View File

@ -1,4 +1,6 @@
//! This program removes all forms of emphasis from the markdown of the book.
//! An example preprocessor for removing all forms of emphasis from a markdown
//! book.
extern crate mdbook;
extern crate pulldown_cmark;
extern crate pulldown_cmark_to_cmark;
@ -6,31 +8,13 @@ extern crate pulldown_cmark_to_cmark;
use mdbook::book::{Book, BookItem, Chapter};
use mdbook::errors::{Error, Result};
use mdbook::preprocess::{Preprocessor, PreprocessorContext};
use mdbook::MDBook;
use pulldown_cmark::{Event, Parser, Tag};
use pulldown_cmark_to_cmark::fmt::cmark;
use std::env::{args, args_os};
use std::ffi::OsString;
use std::process;
const NAME: &str = "md-links-to-html-links";
fn do_it(book: OsString) -> Result<()> {
let mut book = MDBook::load(book)?;
book.with_preprecessor(Deemphasize);
book.build()
}
fn main() {
if args_os().count() != 2 {
eprintln!("USAGE: {} <book>", args().next().expect("executable"));
return;
}
if let Err(e) = do_it(args_os().skip(1).next().expect("one argument")) {
eprintln!("{}", e);
process::exit(1);
}
panic!("This example is intended to be part of a library");
}
struct Deemphasize;

120
examples/nop-preprocessor.rs
View File

@ -1,56 +1,64 @@
extern crate clap;
extern crate mdbook;
extern crate serde_json;
#[macro_use]
extern crate clap;
use clap::{App, Arg, ArgMatches, SubCommand};
use mdbook::preprocess::CmdPreprocessor;
use mdbook::book::Book;
use std::process;
use mdbook::errors::Error;
use mdbook::preprocess::{CmdPreprocessor, Preprocessor, PreprocessorContext};
use std::io;
use std::process;
use nop_lib::Nop;
pub fn make_app() -> App<'static, 'static> {
App::new("nop-preprocessor")
.about("A mdbook preprocessor which does precisely nothing")
.subcommand(
SubCommand::with_name("supports")
.arg(Arg::with_name("renderer").required(true))
.about("Check whether a renderer is supported by this preprocessor"))
}
fn main() {
let matches = app().get_matches();
let matches = make_app().get_matches();
// Users will want to construct their own preprocessor here
let preprocessor = Nop::new();
if let Some(sub_args) = matches.subcommand_matches("supports") {
handle_supports(sub_args);
handle_supports(&preprocessor, sub_args);
} else {
handle_preprocessing();
if let Err(e) = handle_preprocessing(&preprocessor) {
eprintln!("{}", e);
process::exit(1);
}
}
}
fn handle_preprocessing() {
let (ctx, book) = CmdPreprocessor::parse_input(io::stdin())
.expect("Couldn't parse the input");
fn handle_preprocessing(pre: &dyn Preprocessor) -> Result<(), Error> {
let (ctx, book) = CmdPreprocessor::parse_input(io::stdin())?;
// You can inspect the calling mdbook's version to check for compatibility
if ctx.mdbook_version != mdbook::MDBOOK_VERSION {
panic!("The version check failed!");
// We should probably use the `semver` crate to check compatibility
// here...
eprintln!(
"Warning: The {} plugin was built against version {} of mdbook, \
but we're being called from version {}",
pre.name(),
mdbook::MDBOOK_VERSION,
ctx.mdbook_version
);
}
// In testing we want to tell the preprocessor to blow up by setting a
// particular config value
if let Some(table) = ctx.config.get_preprocessor("nop-preprocessor") {
let should_blow_up = table.get("blow-up").is_some();
let processed_book = pre.run(&ctx, book)?;
serde_json::to_writer(io::stdout(), &processed_book)?;
if should_blow_up {
panic!("Boom!!!1!");
}
Ok(())
}
let processed_book = do_processing(book);
serde_json::to_writer(io::stdout(), &processed_book).unwrap();
}
fn do_processing(book: Book) -> Book {
// We *are* a nop preprocessor after all...
book
}
fn handle_supports(sub_args: &ArgMatches) {
fn handle_supports(pre: &dyn Preprocessor, sub_args: &ArgMatches) -> ! {
let renderer = sub_args.value_of("renderer").expect("Required argument");
let supported = renderer_is_supported(&renderer);
let supported = pre.supports_renderer(&renderer);
// Signal whether the renderer is supported by exiting with 1 or 0.
if supported {
@ -60,17 +68,45 @@ fn handle_supports(sub_args: &ArgMatches) {
}
}
fn renderer_is_supported(renderer: &str) -> bool {
// We support everything except the `not-supported` renderer
renderer != "not-supported"
/// The actual implementation of the `Nop` preprocessor. This would usually go
/// in your main `lib.rs` file.
mod nop_lib {
use super::*;
/// A no-op preprocessor.
pub struct Nop;
impl Nop {
pub fn new() -> Nop {
Nop
}
}
fn app() -> App<'static, 'static> {
app_from_crate!().subcommand(
SubCommand::with_name("supports")
.arg(Arg::with_name("renderer").required(true))
.about(
"Check whether a renderer is supported by this preprocessor",
),
)
impl Preprocessor for Nop {
fn name(&self) -> &str {
"nop-preprocessor"
}
fn run(
&self,
ctx: &PreprocessorContext,
book: Book,
) -> Result<Book, Error> {
// In testing we want to tell the preprocessor to blow up by setting a
// particular config value
if let Some(nop_cfg) = ctx.config.get_preprocessor(self.name()) {
if nop_cfg.contains_key("blow-up") {
return Err("Boom!!1!".into());
}
}
// we *are* a no-op preprocessor after all
Ok(book)
}
fn supports_renderer(&self, renderer: &str) -> bool {
renderer != "not-supported"
}
}
}

19
src/preprocess/cmd.rs
View File

@ -3,7 +3,7 @@ use book::Book;
use errors::*;
use serde_json;
use shlex::Shlex;
use std::io::{self, Read};
use std::io::{self, Read, Write};
use std::process::{Child, Command, Stdio};
/// A custom preprocessor which will shell out to a 3rd-party program.
@ -50,23 +50,24 @@ impl CmdPreprocessor {
.chain_err(|| "Unable to parse the input")
}
fn write_input(
fn write_input_to_child(
&self,
child: &mut Child,
book: Book,
book: &Book,
ctx: &PreprocessorContext,
) {
let mut stdin = child.stdin.take().expect("Child has stdin");
let input = (ctx, book);
let stdin = child.stdin.take().expect("Child has stdin");
if let Err(e) = serde_json::to_writer(&mut stdin, &input) {
if let Err(e) = self.write_input(stdin, &book, &ctx) {
// Looks like the backend hung up before we could finish
// sending it the render context. Log the error and keep going
warn!("Error writing the RenderContext to the backend, {}", e);
}
}
// explicitly close the `stdin` file handle
drop(stdin);
fn write_input<W: Write>(&self, writer: W, book: &Book, ctx: &PreprocessorContext) -> Result<()> {
serde_json::to_writer(writer, &(ctx, book))
.map_err(Into::into)
}
/// The command this `Preprocessor` will invoke.
@ -106,7 +107,7 @@ impl Preprocessor for CmdPreprocessor {
.spawn()
.chain_err(|| format!("Unable to start the \"{}\" preprocessor. Is it installed?", self.name()))?;
self.write_input(&mut child, book, ctx);
self.write_input_to_child(&mut child, &book, ctx);
let output = child
.wait_with_output()

108
src/preprocess/links.rs
View File

@ -69,7 +69,12 @@ where
Ok(new_content) => {
if depth < MAX_LINK_NESTED_DEPTH {
if let Some(rel_path) = playpen.link.relative_path(path) {
replaced.push_str(&replace_all(&new_content, rel_path, source, depth + 1));
replaced.push_str(&replace_all(
&new_content,
rel_path,
source,
depth + 1,
));
} else {
replaced.push_str(&new_content);
}
@ -83,6 +88,10 @@ where
}
Err(e) => {
error!("Error updating \"{}\", {}", playpen.link_text, e);
for cause in e.iter().skip(1) {
warn!("Caused By: {}", cause);
}
// This should make sure we include the raw `{{# ... }}` snippet
// in the page content if there are any errors.
previous_end_index = playpen.start_index;
@ -109,10 +118,18 @@ impl<'a> LinkType<'a> {
let base = base.as_ref();
match self {
LinkType::Escaped => None,
LinkType::IncludeRange(p, _) => Some(return_relative_path(base, &p)),
LinkType::IncludeRangeFrom(p, _) => Some(return_relative_path(base, &p)),
LinkType::IncludeRangeTo(p, _) => Some(return_relative_path(base, &p)),
LinkType::IncludeRangeFull(p, _) => Some(return_relative_path(base, &p)),
LinkType::IncludeRange(p, _) => {
Some(return_relative_path(base, &p))
}
LinkType::IncludeRangeFrom(p, _) => {
Some(return_relative_path(base, &p))
}
LinkType::IncludeRangeTo(p, _) => {
Some(return_relative_path(base, &p))
}
LinkType::IncludeRangeFull(p, _) => {
Some(return_relative_path(base, &p))
}
LinkType::Playpen(p, _) => Some(return_relative_path(base, &p)),
}
}
@ -182,11 +199,15 @@ impl<'a> Link<'a> {
match (typ.as_str(), file_arg) {
("include", Some(pth)) => Some(parse_include_path(pth)),
("playpen", Some(pth)) => Some(LinkType::Playpen(pth.into(), props)),
("playpen", Some(pth)) => {
Some(LinkType::Playpen(pth.into(), props))
}
_ => None,
}
}
(Some(mat), None, None) if mat.as_str().starts_with(ESCAPE_CHAR) => {
(Some(mat), None, None)
if mat.as_str().starts_with(ESCAPE_CHAR) =>
{
Some(LinkType::Escaped)
}
_ => None,
@ -207,20 +228,65 @@ impl<'a> Link<'a> {
match self.link {
// omit the escape char
LinkType::Escaped => Ok((&self.link_text[1..]).to_owned()),
LinkType::IncludeRange(ref pat, ref range) => file_to_string(base.join(pat))
LinkType::IncludeRange(ref pat, ref range) => {
let target = base.join(pat);
file_to_string(&target)
.map(|s| take_lines(&s, range.clone()))
.chain_err(|| format!("Could not read file for link {}", self.link_text)),
LinkType::IncludeRangeFrom(ref pat, ref range) => file_to_string(base.join(pat))
.chain_err(|| {
format!(
"Could not read file for link {} ({})",
self.link_text,
target.display(),
)
})
}
LinkType::IncludeRangeFrom(ref pat, ref range) => {
let target = base.join(pat);
file_to_string(&target)
.map(|s| take_lines(&s, range.clone()))
.chain_err(|| format!("Could not read file for link {}", self.link_text)),
LinkType::IncludeRangeTo(ref pat, ref range) => file_to_string(base.join(pat))
.chain_err(|| {
format!(
"Could not read file for link {} ({})",
self.link_text,
target.display(),
)
})
}
LinkType::IncludeRangeTo(ref pat, ref range) => {
let target = base.join(pat);
file_to_string(&target)
.map(|s| take_lines(&s, range.clone()))
.chain_err(|| format!("Could not read file for link {}", self.link_text)),
LinkType::IncludeRangeFull(ref pat, _) => file_to_string(base.join(pat))
.chain_err(|| format!("Could not read file for link {}", self.link_text)),
.chain_err(|| {
format!(
"Could not read file for link {} ({})",
self.link_text,
target.display(),
)
})
}
LinkType::IncludeRangeFull(ref pat, _) => {
let target = base.join(pat);
file_to_string(&target).chain_err(|| {
format!("Could not read file for link {} ({})",
self.link_text,
target.display())
})
}
LinkType::Playpen(ref pat, ref attrs) => {
let contents = file_to_string(base.join(pat))
.chain_err(|| format!("Could not read file for link {}", self.link_text))?;
let target = base.join(pat);
let contents =
file_to_string(&target).chain_err(|| {
format!(
"Could not read file for link {} ({})",
self.link_text,
target.display()
)
})?;
let ftype = if !attrs.is_empty() { "rust," } else { "rust" };
Ok(format!(
"```{}{}\n{}\n```\n",
@ -465,7 +531,10 @@ mod tests {
Link {
start_index: 38,
end_index: 68,
link: LinkType::Playpen(PathBuf::from("file.rs"), vec!["editable"]),
link: LinkType::Playpen(
PathBuf::from("file.rs"),
vec!["editable"]
),
link_text: "{{#playpen file.rs editable }}",
},
Link {
@ -475,7 +544,8 @@ mod tests {
PathBuf::from("my.rs"),
vec!["editable", "no_run", "should_panic"],
),
link_text: "{{#playpen my.rs editable no_run should_panic}}",
link_text:
"{{#playpen my.rs editable no_run should_panic}}",
},
]
);