sztest
/
cdbrelease
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Begin rebuilding to use new CDB API 

Since writing cdbrelease, CDB has added a new set
of stable APIs suitable for creating releases.  Major
changes planned:

- Use the JSON API and token auth for everything
  instead of scraping site HTML.
- Switch to using CDB's standard mechanisms for
  some config, such as *.conf for versions and
  .cdb.json for static settings.
- Switch to zip uploads so we can locally control
  exports, since CDB uses git-export-all which does
  not properly support .gitattributes export-subst.

This also involves a significant rework of the
configuration interfaces to support the new
.cdb.json source, and mirroring the git repo is now
mandatory, as using zip upload, we cannot rely on
CDB to access the repo itself directly.

New configuration interfaces are ostensibly done,
CDB components need to be reworked still.
master
Aaron Suen 2021-02-27 09:31:13 -05:00
parent a1109c2970
commit 8f06601e86
9 changed files with 258 additions and 342 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

63
README
View File

@ -1,24 +1,75 @@
========================================================================
........................................................................
Prerequisites:
- You must have an existing account on content.minetest.net, and must
have a password configured (i.e. you can't use GitHub login for this).
have an API Token configured.
- You must have already setup your package; this script will not create
new packages for you.
- Your package must use VCS for release, and have the source repo URL
already registered; file upload is not supported here.
........................................................................
To install the script:
Installation:
- Have nodejs (and npm if packaged separately) installed.
- Clone/download the project.
- Run "npm i" in the project dir to install dependencies.
........................................................................
Configuration:
Configuration is read from the following sources, from high to low
priority (highest priority overrides all others):
- command line options
- git repository .cdbrelease.lua
- git repository .cdb.json
- local .cdbrelease.json config file
- internal defaults
Note that settings that are used to locate other sources of config (e.g.
the config file path or git repository url) can only be meaningfully
defined in config sources that are available before those sources have
been loaded. Settings become available in the order:
- internal defaults
- command line options
- local .cdbrelease.json config file
- git repository .cdb.json
- git repository .cdbrelease.lua
Recognized settings:
- conf: path to local config file, default' ~/.cdbrelease.json'.
- repo: REQUIRED path or URL to a git repository from which to create
the release; must have access to clone.
- branch: git ref/branch/tag from the source repo to release, defaults
to master.
- execgit: path to execute git command, default 'git' (using $PATH)
- exectar: path to execute tar command, default 'tar' (using $PATH)
- execlua: path to execute lua 5.1 command, default 'lua5.1'
(using $PATH)
- cdbjson: enable this option to enable processing configuration from
the .cdb.json file embedded inside the repo. WARNING: only do this if
you trust the repository; any setting can be defined in the file.
- cdbjsonpath: relative path to .cdb.json within git checkout/export,
default '.cdb.json'.
- cdblua: enable this option to enable processing dynamic lua config
from inside the repo; a lua script embedded in the repo will be
EXECUTED and can return config values. WARNING: only do this if you
trust the repository highly, and you're running adequately sandboxed
(e.g. VM/container); any setting can be defined, and arbitrary code
will be executed.
- cdbluapath: relative path to .cdbrelease.lua within the git
checkout/export, default '.cdbrelease.lua'.
========================================================================
To create a release manually:
- In this project's dir, run:
node . --user="yourname" --pass="yourpass" --pkg="yourpackage" \
node . --user="yourname" --token="yourapitoken" --pkg="yourpackage" \
--version="new-version-title"
- You can also set optional parameters:
@ -41,17 +92,17 @@ To create a release automatically from a git repo:
- Commit (and if necessary push).
- In this project's dir, run:
node . --user="yourname" --pass="yourpass" --fromgit="repourl"
node . --user="yourname" --token="yourapitoken" --repo="repourl"
- "repourl" should be anything that git can clone, including a local
path, a web URL, ssh URL, etc.
- You can also optionally set any parameters listed above in the
"release manually" section. Anything set locally will override the
settings from the repo. Technically you can store your username and
password in the repo, but this is bad practice.
API Token in the repo, but this is bad practice.
N.B. code in .cdbrelease.lua is NOT sandboxed in any way!
There are significant security implications to using --fromgit with a
There are significant security implications to using --repo with a
repo that you do not fully control. Use at your own risk.
========================================================================

88
cdblib.js
View File

@ -1,7 +1,6 @@
'use strict';
const needle = require('needle');
const cheerio = require('cheerio');
const config = require('./config');
const maxattempts = 10;
@ -10,34 +9,25 @@ async function fetch(uri, meth, data, opts, ...rest) {
meth = meth || 'get';
opts = opts || {};
opts.headers = opts.headers || {};
opts.headers.Authorization = `Bearer ${config.token}`;
opts.headers.Referer = uri;
opts.cookies = cookiejar;
if (!opts.hasOwnProperty('follow_max'))
opts.follow_max = 10;
for (let attempt = 1; attempt <= maxattempts; attempt++)
try {
const r = await needle(meth, uri, data, opts, ...rest);
if (r.cookies)
Object.keys(r.cookies)
.forEach(k => cookiejar[k] = r.cookies[k]);
const $ = cheerio.load(r.body.toString());
let oops;
$('h1')
.each(function () {
const t = $(this)
.text();
if (/^\s*Oops!/.test(t))
oops = t;
});
if (oops || !/^[123]\d\d/.test(r.statusCode.toString())) {
const resp = await needle(meth, uri, data, opts, ...rest);
if (resp.cookies)
Object.keys(resp.cookies)
.forEach(k => cookiejar[k] = resp.cookies[k]);
if (!/^[23]\d\d/.test(resp.statusCode.toString())) {
const failure = {
statusCode: r.statusCode,
statusMessage: r.statusMessage
statusCode: resp.statusCode,
statusMessage: resp.statusMessage
};
if (oops) failure.oops = oops;
throw (Object.assign(new Error(JSON.stringify(failure)), failure));
}
return $;
return resp;
}
catch (err) {
if (attempt === maxattempts
@ -48,62 +38,4 @@ async function fetch(uri, meth, data, opts, ...rest) {
}
}
function getfields($) {
const fields = {};
$('input')
.each(function () {
const e = $(this);
const n = e.attr('name');
if (!n) return;
const v = e.attr('value');
if (v) fields[n] = v;
});
return fields;
}
async function login_core() {
const uri = config.root + '/user/login/';
console.log('fetching login page...');
let $ = await fetch(uri);
const fields = getfields($);
fields.username = config.user;
fields.password = config.pass;
console.log('submitting login form...');
$ = await fetch(uri, 'post', fields, { follow_max: 0 });
let success;
$('h1')
.each(function () {
success = success || /^\s*Redirecting\b/i.test($(this)
.text());
});
if (!success)
throw Error('login no redirect: ' + $('body')
.text());
}
let loginprom;
async function login() {
if (!loginprom)
loginprom = login_core();
return await loginprom;
}
function findopt($, sel, name) {
let v;
$(sel)
.each(function () {
const o = $(this);
const t = o.text();
if (t.toLowerCase()
.startsWith(name.toLowerCase()))
v = o.attr('value');
});
return v;
}
module.exports = {
fetch,
getfields,
login,
findopt
};
module.exports = fetch;

12
config.js
View File

@ -1,13 +1,13 @@
'use strict';
const process = require('process');
const path = require('path');
const minimist = require('minimist');
const config = {};
module.exports = config;
const layers = {};
const layord = 'default fromfile modscan fromgit cmdline'.split(' ');
const layord = 'default conffile gitjson gitlua cmdline'.split(' ');
function set(name, obj) {
if(!layord.find(x => x === name))
@ -24,13 +24,17 @@ function set(name, obj) {
set('cmdline', minimist(process.argv.slice(2)));
set('default', {
conf: path.join(process.env.HOME, '.cdbrelease.json'),
branch: 'master',
execlua: 'lua5.1',
cdbjsonpath: '.cdb.json',
cdbluapath: '.cdbrelease.lua',
root: 'https://content.minetest.net/',
token: null,
user: null,
pass: null,
pkg: null,
min: 'none',
max: 'none',
branch: 'master',
timeout: 60,
poll: 2
});

138
fromgit.js
View File

@ -1,138 +0,0 @@
'use strict';
const fs = require('fs');
const path = require('path');
const util = require('util');
const child = require('child_process');
const tmp = require('tmp-promise');
const fsx = require('fs-extra');
const config = require('./config');
const readFile = util.promisify(fs.readFile);
function luaser(obj) {
if(Array.isArray(obj))
return `{${obj.map(luaser).join(',')}}`;
if(!!obj && typeof obj === 'object' && obj.toString() === '[object Object]')
return `{${Object.keys(obj)
.map(k => ({
k: /^[A-Za-z_][A-Za-z0-9_]*$/.test(k) ? k : JSON.stringify(k),
v: luaser(obj[k])
}))
.filter(x => x.v !== 'nil')
.map(x => x.k + '=' + x.v)}}`;
if(typeof obj === 'number' || typeof obj === 'boolean')
return `${obj}`;
if(typeof obj === 'string' || obj instanceof String)
return JSON.stringify(obj);
if(!obj || typeof obj === 'function')
return 'nil';
return JSON.stringify(obj.toString());
}
function spawn(path, params, ...args) {
console.log(`> ${path} ${params.join(' ')}`);
const proc = child.spawn(path, params, ...args);
proc.promise = new Promise((res, rejraw) => {
function rej(e) {
rejraw(`${path} ${params.join(' ')}: ${e}`);
}
proc.on('error', rej);
proc.on('exit', (code, sig) => {
if(sig) rej('signal: ' + sig);
if(code !== 0) rej('code: ' + code);
res();
});
});
return proc;
}
async function gitexport(repodir, outdir) {
const git = config.execgit || 'git';
const tar = config.exectar || 'tar';
const garch = spawn(git, [
'--git-dir=' + repodir,
'archive',
'--format=tar',
config.branch
], {
stdio: ['inherit', 'pipe', 'inherit']
});
const untar = spawn(tar, [
'-C', outdir,
'-xf', '-'
], {
stdio: ['pipe', 'inherit', 'inherit']
});
garch.stdout.pipe(untar.stdin);
await garch.promise;
await untar.promise;
}
module.exports = async () => {
const luahook = `config = ${luaser(config)}\n\n` +
(await readFile('luahook.lua'))
.toString();
const tmpdir = await tmp.dir({
mode: parseInt('700', 8),
prefix: 'cdbrelease-'
});
try {
const git = config.execgit || 'git';
const lua = config.execlua || 'lua5.1';
console.log('cloning source git repo...');
const repodir = path.join(tmpdir.path, 'repo');
await spawn(git, [
'clone',
'--bare',
'--depth=1',
'-b', config.branch,
config.fromgit,
repodir
], {
stdio: 'inherit'
})
.promise;
console.log('building full export for metadata...');
const fulldir = path.join(tmpdir.path, 'full');
await fsx.emptyDir(fulldir);
await spawn(git, [
'clone',
repodir,
fulldir
])
.promise;
await gitexport(repodir, fulldir);
console.log('loading metadata...')
let buff = '';
const hook = spawn(lua, ['-'], {
stdio: ['pipe', 'pipe', 'inherit'],
cwd: fulldir
});
hook.stdout.on('data', x => buff += x.toString());
hook.stdin.write(luahook);
hook.stdin.end();
await hook.promise;
const cleanfull = fsx.emptyDir(fulldir);
const spec = JSON.parse(buff);
config.set('fromgit', spec);
await cleanfull;
console.log('building standard export for mod scan...');
const expodir = path.join(tmpdir.path, 'export');
await fsx.emptyDir(expodir);
await gitexport(repodir, expodir);
console.log('scanning included mods...');
const scanned = await require('./modscan')(expodir);
config.set('modscan', scanned);
} finally {
await fsx.emptyDir(tmpdir.path);
await tmpdir.cleanup();
}
};

48
gitlua.js Normal file
View File

@ -0,0 +1,48 @@
'use strict';
const fs = require('fs');
const util = require('util');
const config = require('./config');
const spawn = require('./spawn');
const readFile = util.promisify(fs.readFile);
function luaser(obj) {
if(Array.isArray(obj))
return `{${obj.map(luaser).join(',')}}`;
if(!!obj && typeof obj === 'object' && obj.toString() === '[object Object]')
return `{${Object.keys(obj)
.map(k => ({
k: /^[A-Za-z_][A-Za-z0-9_]*$/.test(k) ? k : JSON.stringify(k),
v: luaser(obj[k])
}))
.filter(x => x.v !== 'nil')
.map(x => x.k + '=' + x.v)}}`;
if(typeof obj === 'number' || typeof obj === 'boolean')
return `${obj}`;
if(typeof obj === 'string' || obj instanceof String)
return JSON.stringify(obj);
if(!obj || typeof obj === 'function')
return 'nil';
return JSON.stringify(obj.toString());
}
module.exports = async cwd => {
const luahook = `config = ${luaser(config)}\n\n` +
(await readFile('luahook.lua'))
.toString();
console.log('loading metadata...');
let buff = '';
const hook = spawn('lua', ['-'], {
stdio: ['pipe', 'pipe', 'inherit'],
cwd
});
hook.stdout.on('data', x => buff += x.toString());
hook.stdin.write(luahook);
hook.stdin.end();
await hook.promise;
const spec = JSON.parse(buff);
config.set('gitlua', spec);
};

74
gitmirror.js Normal file
View File

@ -0,0 +1,74 @@
'use strict';
const fsx = require('fs-extra');
const tmp = require('tmp-promise');
const config = require('./config');
const spawn = require('./spawn');
async function withtemp(func) {
const tmpdir = await tmp.dir({
mode: parseInt('700', 8),
prefix: 'cdbrelease-'
});
try {
await fsx.emptyDir(tmpdir.path);
return await func(tmpdir.path);
} finally {
await fsx.emptyDir(tmpdir.path);
await tmpdir.cleanup();
}
}
async function rawexport(repodir, outdir) {
const garch = spawn('git', [
'--git-dir=' + repodir,
'archive',
'--format=tar',
config.branch
], {
stdio: ['inherit', 'pipe', 'inherit']
});
const untar = spawn('tar', [
'-C', outdir,
'-xf', '-'
], {
stdio: ['pipe', 'inherit', 'inherit']
});
garch.stdout.pipe(untar.stdin);
await garch.promise;
await untar.promise;
}
async function gitmirror(func) {
return await withtemp(async tmp => {
await spawn('git', [
'clone',
'--bare',
'--depth=1',
'-b', config.branch,
config.repo,
tmp
], {
stdio: 'inherit'
})
.promise;
async function gitexport(full, f2) {
return withtemp(async t2 => {
if(full)
await spawn('git', [
'clone',
'-b', config.branch,
tmp,
t2
])
.promise;
await rawexport(tmp, t2);
return await f2(t2);
});
}
gitexport.path = tmp;
return await func(gitexport);
});
}
module.exports = gitmirror;

64
index.js
View File

@ -1,34 +1,44 @@
'use strict';
process.on('unhandledRejection', e => { throw e; });
const fs = require('fs');
const process = require('process');
const path = require('path');
const util = require('util');
const path = require('path');
const config = require('./config');
const fromgit = require('./fromgit');
const cdbrelease = require('./cdbrelease');
const cdbedit = require('./cdbedit');
const cdbscreens = require('./cdbscreens');
// const cdbrelease = require('./cdbrelease');
// const cdbedit = require('./cdbedit');
// const cdbscreens = require('./cdbscreens');
const gitmirror = require('./gitmirror');
const gitlua = require('./gitlua');
const readFile = util.promisify(fs.readFile);
process.on('unhandledRejection', e => {
console.warn(e);
process.exit(1);
});
(async () => {
const loadJsonConfig = async (fn, key, req) => {
try {
const conf = await readFile(path.join(process.env.HOME,
'.cdbrelease.json'));
if(conf)
config.set('fromfile', JSON.parse(conf.toString()));
const conf = await readFile(fn);
config.set(key, JSON.parse(conf.toString()));
} catch (e) {
if(req) throw e;
console.warn(e.message || e);
}
}
if(config.fromgit)
await fromgit();
(async () => {
await loadJsonConfig(config.conf, 'conffile');
if(!config.repo)
throw 'missing/invalid option: --repo=...';
await gitmirror(async gitexport => {
if(config.cdbjson || config.cdblua)
await gitexport(true, async full => {
if(config.cdbjson)
await loadJsonConfig(
path.join(full, config.cdbjsonpath),
'gitjson', true);
if(config.cdblua)
await gitlua(full);
});
});
const missing = [];
Object.keys(config)
@ -43,12 +53,12 @@ process.on('unhandledRejection', e => {
throw 'missing/invalid options: ' + missing.join(' ');
config.root = config.root.replace(/\/+$/, '');
if(config.version) {
config.version = '' + config.version;
config.edit = await cdbrelease() || config.edit;
}
if(config.edit)
config.uploadimages = await cdbedit() || config.uploadimages;
if(config.uploadimages && config.screenshots)
await cdbscreens();
// if(config.version) {
// config.version = '' + config.version;
// config.edit = await cdbrelease() || config.edit;
// }
// if(config.edit)
// config.uploadimages = await cdbedit() || config.uploadimages;
// if(config.uploadimages && config.screenshots)
// await cdbscreens();
})();

89
modscan.js
View File

@ -1,89 +0,0 @@
const fs = require('fs');
const path = require('path');
const config = require('./config');
async function readstr(...paths) {
return await new Promise((r, j) => fs.readFile(path.join(...paths), (e, x) => {
if(e && e.code === 'ENOENT') return r();
if(e) return j(e);
return r(x.toString());
}));
}
async function findmods(data, ...pathparts) {
const pathname = path.join(...pathparts);
const ents = {};
(await new Promise(r => fs.readdir(pathname, (e, x) => e ? r([]) : r(x))))
.filter(x => !x.startsWith('.'))
.forEach(x => ents[x] = true);
let recurse = Object.keys(ents);
if(ents['game.conf'])
recurse = recurse.filter(x => x === 'mods');
const tasks = recurse.map(x => findmods(data, ...pathparts, x));
let modname = (pathparts.length === 1) ? config.pkg : pathparts[pathparts.length - 1];
if(ents['depends.txt'])
(await readstr(pathname, 'depends.txt') || '')
.split('\n')
.filter(x => /\S/.test(x))
.map(x => x.replace(/#.*/, '')
.trim())
.forEach(x => {
if(x.endsWith('?'))
data.soft[x.replace('?', '')] = true;
else
data.hard[x] = true;
});
if(ents['mod.conf']) {
const conf = {};
(await readstr(pathname, 'mod.conf') || '')
.split('\n')
.map(x => x.replace(/#.*/, ''))
.forEach(x => {
const a = x.split('=', 2)
.map(x => x.trim());
conf[a[0]] = a[1];
});
if(conf.depends)
conf.depends
.split(',')
.map(x => x.trim())
.forEach(x => data.hard[x] = true);
if(conf.optional_depends)
conf.optional_depends
.split(',')
.map(x => x.trim())
.forEach(x => data.soft[x] = true);
if(conf.name)
modname = conf.name;
}
if(ents['init.lua'])
data.prov[modname] = true;
await Promise.all(tasks);
}
function apply(obj, key, src) {
if(src && Object.keys(src).length)
obj[key] = Object.keys(src)
.sort()
.join(',');
}
async function modscan(pathname) {
const data = {
prov: {},
hard: {},
soft: {}
};
await findmods(data, pathname);
const meta = {};
apply(meta, 'provides_str', data.prov);
apply(meta, 'harddep_str', data.hard);
apply(meta, 'softdep_str', data.soft);
return meta;
}
module.exports = modscan;

24
spawn.js Normal file
View File

@ -0,0 +1,24 @@
'use strict';
const childproc = require('child_process');
const config = require('./config');
function spawn(path, params, ...args) {
path = config[`exec${path}`] || path;
console.log(`> ${path} ${params.join(' ')}`);
const proc = childproc.spawn(path, params, ...args);
proc.promise = new Promise((res, rejraw) => {
function rej(e) {
rejraw(`${path} ${params.join(' ')}: ${e}`);
}
proc.on('error', rej);
proc.on('exit', (code, sig) => {
if(sig) rej('signal: ' + sig);
if(code !== 0) rej('code: ' + code);
res();
});
});
return proc;
}
module.exports = spawn;