Ref/meta!
Featured Article
How to Code a Simple Media Manager & File Uploader for a Node/Express Site in 2022
By AstroMacGuffin dated Wed Jul 27 2022 22:06:06 GMT+0000 (Coordinated Universal Time) last updated Sat Sep 03 2022 22:39:09 GMT+0000 (Coordinated Universal Time) The first thing you need to know about this media manager is that it's not a good idea to expose it to normal users. They'll be able to create folders, and the filename screening isn't good enough to trust the inputs from random people.
But if you need a way to get Markdown-formatted image tags easily from the images on your website, or just a working version of Express.js file uploads in 2022, as I did, this lesson will get you there. And if you need a springboard you can bring your own improvements to, as I've done since this article was first pushed... I enjoy working with it. I hope you will too.
### Ingredients
- a simple `MediaManager` object on the server side, to handle utilities such as setting/getting path names, creating folders, and getting lists of folders and files in a given subfolder
- a POST route in your Express.js app for handling all MediaManager requests (which will all be AJAX, via jQuery)
- middleware that acts on your route - in the form of an NPM package, called `multer`
- a pug template for the media manager interface
- some extra pug code in whatever interfaces where you want to enable the media manager
- jQuery and a plugin for jQuery, called jquery-uploadfile, which takes a ton of sweat out of the process and gives you a great UI experience including progress bars and options for multiple file uploads and drag 'n drop
- about 145 lines of CSS code
- a client-side JavaScript file to set up the media manager's interface buttons and so forth
### What This Media Manager Does
It's basically a file browser. You can navigate folders, and when you find the image you want, you click it; it then copies Markdown code to your clipboard which you can paste into an input or (more likely) a textarea.
It handles file uploads. The files will be uploaded to the folder you're viewing. It also creates new folders.
### What This Media Manager Doesn't Do
It can't delete files. I decided that, since this is a tool only for admins, and I'm going to be the only admin of this site for the foreseeable future, I can delete files another way.
### On To The Code! First Up: The Server Side Object
This is `media-manager.js`, a Node object:
```js
const path = require('path');
const fs = require('fs');
class MediaManager {
constructor(pathName=global.config.media_path) {
if (pathName.indexOf('..') === -1 && pathName.search(/^\//) === -1)
this.basePath = path.join(__dirname, pathName);
else
this.basePath = path.join(__dirname, global.config.media_path);
this.subPath = '';
}
setSubPath(pathName) {
if (pathName.indexOf('..') > -1 || pathName.search(/^\//) > -1)
return false;
this.subPath = pathName;
return true;
}
getPath() {
return path.join(__dirname, global.config.media_path, this.subPath);
}
createFolder(pathName) {
if (pathName.indexOf('..') > -1 || pathName.search(/^\//) > -1)
return false;
let fullPath = path.join(this.basePath, this.subPath, pathName);
let r = fs.mkdirSync(fullPath, { recursive: true });
if (r === undefined) return false;
return true;
}
getFolders() {
let pathName = path.join(this.basePath, this.subPath);
return fs.readdirSync(pathName, { withFileTypes: true })
.filter(f => f.isDirectory())
.map(f => f.name)
}
getFiles() {
let pathName = path.join(this.basePath, this.subPath);
return fs.readdirSync(pathName, { withFileTypes: true })
.filter(f => !f.isDirectory())
.map(f => f.name)
}
}
module.exports = MediaManager;
```
Let's break that down:
```js
const path = require('path');
const fs = require('fs');
```
We'll be merging paths and doing disk operations, so we need the above packages.
```js
class MediaManager {
constructor(pathName=global.config.media_path) {
if (pathName.indexOf('..') === -1 && pathName.search(/^\//) === -1)
this.basePath = path.join(__dirname, pathName);
else
this.basePath = path.join(__dirname, global.config.media_path);
this.subPath = '';
}
```
The `pathName` argument for the constructor is optional; my code has a `global.config` object with a `media_path` property which is a relative path from the website's root folder (where the app script resides). The above code does some basic security steps, making sure the input doesn't attempt to go to the parent directory or start at the root directory.
```js
setSubPath(pathName) {
if (pathName.indexOf('..') > -1 || pathName.search(/^\//) > -1)
return false;
this.subPath = pathName;
return true;
}
```
With the same security steps as the constructor, the `setSubPath()` method, shown above, stores a property so that it can be merged later with the `basePath` property and `__dirname` (this file is in the project root folder, so `__dirname` will return that folder). This is how we 'select' a path with the object.
```js
getPath() {
return path.join(__dirname, global.config.media_path, this.subPath);
}
```
The above is a utility function that gets me the full, absolute path to the currently selected subfolder.
```js
createFolder(pathName) {
if (pathName.indexOf('..') > -1 || pathName.search(/^\//) > -1)
return false;
let fullPath = path.join(this.basePath, this.subPath, pathName);
let r = fs.mkdirSync(fullPath);
if (r === undefined) return false;
return true;
}
```
Again the same security steps. We then create a folder, returning `true` on success and `false` on failure.
```js
getFolders() {
let pathName = path.join(this.basePath, this.subPath);
return fs.readdirSync(pathName, { withFileTypes: true })
.filter(f => f.isDirectory())
.map(f => f.name)
}
```
The above method, `getFolders()`, returns an array of folder names found in the current path selected by the object.
```js
getFiles() {
let pathName = path.join(this.basePath, this.subPath);
return fs.readdirSync(pathName, { withFileTypes: true })
.filter(f => !f.isDirectory())
.map(f => f.name)
}
```
The `getFiles()` method is the same as `getFolders()` (actually identical except for an exclamation mark), but it returns an array of files in the selected folder.
```js
}
module.exports = MediaManager;
```
We end the class and export it.
### Next: multer and the POST Route
First, you need to install `multer` using this command in the terminal:
```
npm install multer
```
Then you need a route that takes advantage of that feature. At the top of the router file (or the top of your main app file, if you aren't using router files), you'll need this:
```js
const multer = require('multer');
const upload = multer({ dest: global.config.media_path });
const MediaManager = require('../media-manager');
```
Note above that my `media-manager.js` file is in the parent directory compared to where my router files sit. Your project structure may be different.
Now we get into the actual route that handles all operations from the media manager:
```js
router.post(
'/media-manager',
upload.single('uploadedFile'),
async (req, res) => {
if (req.session.isAdmin === undefined || !req.session.isAdmin) {
return res.redirect('/user');
}
let m = new MediaManager();
let arg = (req.method === 'POST') ? req.body : req.params;
let r = {};
switch (arg.cmd) {
case 'showDir':
if (m.setSubPath(arg.subPath)) {
r = {
folders: m.getFolders(),
files: m.getFiles(),
}
}
else {
r = { status: false }
}
break;
case 'upload':
r = {};
if (req.file === undefined) {
r = { status: false };
}
else if (!m.setSubPath(arg.subPath)) {
r = { status: false };
}
else {
let file = req.file;
let oldPath = path.join(__dirname, '..', file.path);
let newPath = path.join(m.getPath(), file.originalname);
fs.renameSync(oldPath, newPath, (err) => {
if (err) r = { status: false };
else r = { status: true };
});
}
break;
case 'mkDir':
if (m.setSubPath(arg.subPath) && m.createFolder(arg.newFolder)) {
r = { status: true };
}
else {
r = { status: false };
}
break;
}
res.json(r);
}
);
```
Let's break that down:
```js
router.post(
'/media-manager',
upload.single('uploadedFile'),
async (req, res) => {
```
The above says: if a POST request is made to the `/media-manager` URL, invoke the `upload` middleware to allow a single upload; we're expecting our file to have the form element name `uploadedFile`. And of course, we create a callback function to handle this route.
```js
if (req.session.isAdmin === undefined || !req.session.isAdmin) {
return res.redirect('/user');
}
```
If the user isn't an admin, we redirect them to the login/register page, even if they're already logged in. It strikes me as a good hint ;)
```js
let m = new MediaManager();
```
We create a new MediaManager object called `m`.
```js
let arg = (req.method === 'POST') ? req.body : req.params;
```
Originally this was a `.all` route so I wasn't sure which source the form fields would be found in, `req.body` or `req.params`. The above handles that, moving the form body variables into the `arg` variable.
```js
let r = {};
```
The variable `r` will be our return value.
```js
switch (arg.cmd) {
```
As I said, every action the media manager is capable of, will be handled by this one route. The `arg.cmd` form field (a hidden field, or in some cases just a JavaScript object on the browser side) tells us what operation we're being asked to perform on the server side.
```js
case 'showDir':
if (m.setSubPath(arg.subPath)) {
r = {
folders: m.getFolders(),
files: m.getFiles(),
}
}
else {
r = { status: false }
}
break;
```
If the value of `arg.cmd` is equal to `"showDir"` then we do this code. The `MediaManager` object, `m`, has three methods we're using here. The `m.setSubPath()` method selects which subfolder the user wants to see. (If `arg.subPath` is an empty string, `MediaManager` just stays in the folder it was configured to treat as the root folder.) Then we pack `r`, our return value, with a `folders` field populated by `m.getFolders()` and a `files` field populated by `m.getFiles()`. If for any reason `m.setSubPath()` returns false -- namely security reasons -- then instead the return value becomes an object containing only `status: false`.
```js
case 'upload':
r = {};
if (req.file === undefined) {
r = { status: false };
}
else if (!m.setSubPath(arg.subPath)) {
r = { status: false };
}
else {
let file = req.file;
let oldPath = path.join(__dirname, '..', file.path);
let newPath = path.join(m.getPath(), file.originalname);
fs.renameSync(oldPath, newPath, (err) => {
if (err) r = { status: false };
else r = { status: true };
});
}
break;
```
Thank goodness for that middleware, `multer`. Thanks to it, `req.file` contains our uploaded file data. (Multer can also handle multiple file uploads at once, in which case `req.files` would instead be an array of files rather than `req.file` being a file -- but let's keep it simple for now.) Above, we do two screening steps: if `req.file` is undefined or if we can't select the desired subfolder with `m.setSubPath()` then we set our return value to `{status:false}` and move along home.
But if both of those checks pass, we reach the `else` statement, above, which handles the actual upload. We compose the `oldPath` and `newPath` variables, and use `fs.renameSync()` to move the file from its default location to our desired destination. Notice that, since this router file is in a subfolder of my main project folder, I had to put `'..'` into the `path.join()` call after `__dirname` (which returns the absolute path to the directory where the current file is found). Our return value, `r`, is given a `status` field depending on the success or failure of the `fs.renameSync()` call, and we're done with the file upload process.
```js
case 'mkDir':
if (m.setSubPath(arg.subPath) && m.createFolder(arg.newFolder)) {
r = { status: true };
}
else {
r = { status: false };
}
break;
```
Thanks to my node object `m` (an instance of the `MediaManager` class) the command to make a folder is very simple. Both `m.setSubPath()` and `m.createFolder()` return true or false depending on success, so we can do them both in the `if` statement, connected with `&&` (meaning "logical and", i.e. "if both of these conditions evaluate to `true`"). The functions happen regardless of whether they evaluate to `true` or `false`, except that the 2nd function call (`m.createFolder()`) won't happen if the first one (`m.setSubPath()`) returns false -- a feature called short circuiting.
Oh, and in case you missed it during all that jargon: we've created the folder. If both those functions return true, we set the return value `r` to `{status:true}` and move along home.
```js
}
res.json(r);
}
);
```
This concludes the `switch` and the route handler. We send `r` to the browser with JSON response headers and we're done.
### The Pug Template
This part is easy and requires no explanation if you've used Pug before. Here is `mm.pug` in my `views` folder:
```
div#media_manager
span#close_x X
h3 Folder:
span#folderNameDisplay
div#folder_list
- // populated on client-side
h3 Upload a File
div#upload_form(enctype="multipart/form-data")
#fileuploader Upload
input#subPath(type="hidden", name="subPath", value="")
input#cmd(type="hidden", name="cmd", value="upload")
h3 Create a Folder
form#create_folder
label(for="newFolder") New Folder name:
input#new_folder_name(type="text", name="newFolder", value="")
input#new_folder_submit(type="button", name="submit", value="Create Folder")
```
Of course, the CSS and browser-side JavaScript will be referencing those classes and ID's, so be thorough if you decide to change any of them.
### Pug Code Wherever You Need the Media Manager to Appear
There are two insertions to make in whatever Pug template you want to add the media manager to: a button to make the media manager appear, and an include to invoke the media manager's template.
The button code:
```
div.content_buttons
span.open_mm 🖼️ Media Manager
span.copy_msg Markdown img copied. Now paste it into the content below.
```
The inclusion code:
```
if isAdmin
include mm
script(src="/static/media-manager.js")
```
Every route on my app sends the `isAdmin` variable to the template. Make adjustments for whatever security measure you're using. Also remember indentation is key with Pug templates, so don't blindly assume my indentations in the two snippets above will make sense for your template.
### jQuery and jQuery-uploadfile
In addition to jQuery I'm also using jQuery-UI. Here is the Pug code to include the lot, assuming you have downloaded the packages to your `static` folder:
```
script(src="/static/jquery-ui/external/jquery/jquery.js")
link(rel="stylesheet", href="/static/jquery-ui/jquery-ui.structure.min.css", type="text/css")
link(rel="stylesheet", href="/static/jquery-ui/jquery-ui.min.css", type="text/css")
link(rel="stylesheet", href="/static/jquery-ui/jquery-ui.theme.min.css", type="text/css")
script(src="/static/jquery-ui/jquery-ui.min.js")
script(src="/static/jquery.uploadfile.min.js")
```
### The CSS
I'm not prepared to teach you CSS today; I'll just include the CSS code you need, and encourage you to search W3Schools for the things you're unfamiliar with. Learning how to code is superior to copy-pasting blindly.
```css
/* Media Manager ************************************************************ */
#media_manager {
height: 97vh;
width: 97vw;
position: fixed;
top: 1.5vw;
left: .5vh;
z-index: 4;
background-color: #222;
display: none;
border-radius: 1em;
border: 5px solid #f77a04;
box-sizing: content-box;
}
#media_manager h3 {
margin-left: 10vw;
}
#folderNameDisplay {
margin-left: 25px;
}
.open_mm, #close_x {
cursor: pointer;
}
#media_manager #close_x {
position: absolute;
color: red;
background-color: black;
right: 1em;
top: 1em;
width: 50px;
height: 50px;
font-size: 40px;
font-weight: bold;
padding: 5px;
text-align: center;
}
#media_manager #close_x:hover {
background: red;
color: black;
}
#media_manager #folder_list {
width: 66vw;
height: 55vh;
margin: auto;
padding: 10px;
background-color: #111;
color: #dcdcdc;
border: 1px solid #666;
overflow-y: scroll;
}
#media_manager #folder_list::-webkit-scrollbar,
#media_manager #upload_form::-webkit-scrollbar {
width: 12px; /* width of the entire scrollbar */
}
#media_manager #folder_list::-webkit-scrollbar-track,
#media_manager #upload_form::-webkit-scrollbar-track {
background: #222; /* color of the tracking area */
}
#media_manager #folder_list::-webkit-scrollbar-thumb,
#media_manager #upload_form::-webkit-scrollbar-thumb {
background-color: #444; /* color of the scroll thumb */
border-radius: 20px; /* roundness of the scroll thumb */
border: 3px solid #444; /* creates padding around scroll thumb */
}
#media_manager #upload_form {
width: 66vw;
height: 10vh;
margin: auto;
padding: 10px;
border: 1px solid #666;
}
#media_manager #create_folder {
width: 66vw;
padding: 10px;
margin: auto;
}
.mm_icon_form {
display: inline-block;
vertical-align: top;
margin: 10px 10px;
}
.mm_icon_image {
width: 64px;
height: 64px;
background-color: transparent;
border: 1px solid #333;
font-size: 2em;
}
.mm_icon_form > p {
width: 50px;
font-size: 11pt;
vertical-align: top;
text-align: center;
margin: 0 0;
word-wrap: break-word;
}
.copy_msg {
margin-left: 10px;
display: none;
font-weight: bold;
}
#upload_form {
overflow-y: scroll;
}
#fileuploader {
float: left;
display: inline-block;
}
.ajax-file-upload-statusbar {
border: none;
}
#new_folder_name {
background-color: transparent;
color: #dcdcdc;
padding: 10px;
border: 1px solid #666;
width: 40vw;
margin: auto 15px;
}
#new_folder_submit {
border-radius: 3px;
border: 0px solid white;
}
#upload_form #fileuploader .ajax-file-upload, #new_folder_submit {
color: #dcdcdc;
background-color: rgb(24, 69, 93);
font-family: 'Roboto Condensed';
font-size: 13pt;
padding: 10px;
font-weight: bold;
line-height: 13pt !important;
height: auto;
cursor: pointer;
}
#upload_form #fileuploader .ajax-file-upload form input[type="file"] {
height: 100%;
}
.ajax-file-upload-container {
margin-top: 0;
}
.ajax-file-upload-statusbar {
margin-top: 0;
padding-top: 0;
}
```
### Finally, the Browser-Side JavaScript
```js
var dirContents = {};
function renderFolderContents() {
let folders = files = [];
$('#folder_list').html('');
if (dirContents.folders) folders = dirContents.folders;
if (dirContents.files) files = dirContents.files;
// if subfolder exists make ".." link
let subPath = $('#subPath').val();
let backPath = '';
if (subPath !== '') {
let subArray = subPath.split('/');
if (subArray.length > 1) {
backPath = subArray.splice(subArray.length-1).join('/');
}
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('name', 'back');
i.setAttribute('value', '⬅️');
i.setAttribute('id', '..');
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = 'Go back';
f.appendChild(p);
$('#folder_list').append(f);
document.getElementById('..').addEventListener('click', (e) => {
console.log(e.target.id);
$('#folderNameDisplay').html(
(backPath === '') ? 'Root folder' : backPath
);
$('#subPath').val(backPath);
$.post(
'/media-manager',
{
cmd: 'showDir',
subPath: $('#subPath').val(),
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
}).fail(() => {
})
.always(() => {
});
e.preventDefault();
});
}
folders.forEach((item) => {
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('name', item);
i.setAttribute('value', '📁');
i.setAttribute('id', item);
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = item;
f.appendChild(p);
$('#folder_list').append(f);
document.getElementById(item).addEventListener('click', (e) => {
console.log(e.target.id);
let val = (subPath !== '')
? `${subPath}/${e.target.id}`
: e.target.id;
$('#folderNameDisplay').html(val);
$('#subPath').val(val);
$.post(
'/media-manager',
{
cmd: 'showDir',
subPath: $('#subPath').val(),
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
}).fail(() => {
})
.always(() => {
});
e.preventDefault();
})
});
files.forEach((item) => {
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('value', '');
let sub = ($('#subPath').val()) ? `${$('#subPath').val()}/` : '';
let url = `/static/img/mm/${sub}${item}`;
i.setAttribute('id', item);
i.style.backgroundImage = `url('${url}')`;
i.style.backgroundSize = 'cover';
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = item;
f.appendChild(p);
$('#folder_list').append(f);
document.getElementById(item).addEventListener('click', (e) => {
console.log(e.target.id);
navigator.clipboard.writeText(
``
);
$('#media_manager').fadeOut(150);
$('.copy_msg').fadeIn(1000).fadeOut(5000);
e.preventDefault();
})
});
}
$(document).ready(async () => {
// Media manager always starts at the root folder
$('#folderNameDisplay').html('Root folder');
// initialize jquery-uploadfile plugin
$("#fileuploader").uploadFile({
url: "/media-manager",
fileName: 'uploadedFile',
multiple: false,
dragDrop: false,
dynamicFormData: () => {
return { cmd: $('#cmd').val(), subPath: $('#subPath').val(), };
},
returnType: 'json',
onSuccess: (files,data,xhr,pd) => {
console.log(JSON.stringify(data));
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
},
});
// setup create folder button
document.getElementById('create_folder')
.addEventListener('submit', (e) => {
let newFolder = $('#new_folder_name').val();
if (
newFolder === ''
|| newFolder.indexOf('/') > -1
|| newFolder.indexOf('\\') > -1
) return false;
$.post(
`/media-manager`,
{
cmd: 'mkDir', subPath: $('#subPath').val(),
newFolder: newFolder,
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
console.log(data);
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
})
.fail(() => {
})
.always(() => {
});
e.preventDefault();
});
// get initial folder data
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
$('#close_x').click(() => {
$('#media_manager').fadeOut(150);
});
$('.open_mm').click(() => {
$('#media_manager').fadeIn(150);
});
});
```
You may notice some blank space and think to yourself, that's unlike Astro. Most of it is blanks you can fill in with error-trapping code and other trigger code. I felt I'd be remiss if I removed them, since you might not know the jQuery API supports easily chaining all those options off a single AJAX call.
Let's break it down:
```js
var dirContents = {};
```
Every time we make an AJAX call to refresh the contents of a folder, that data goes in `dirContents` before we call `renderFolderContents()`.
```js
function renderFolderContents() {
```
This is a fairly long function, but also fairly simple.
```js
let folders = files = [];
```
Initialize two variables we'll be looping through later.
```js
$('#folder_list').html('');
```
The above resets the contents of the HTML/DOM element with the id `folder_list`, to blank.
```js
if (dirContents.folders) folders = dirContents.folders;
if (dirContents.files) files = dirContents.files;
```
The `dirContents` object should contain a `folders` field and a `files` field before we hit this function. We're really just making shortcut variables here.
```js
// if we're in a subfolder, make ".." link
let subPath = $('#subPath').val();
let backPath = '';
if (subPath !== '') {
let subArray = subPath.split('/');
if (subArray.length > 1) {
backPath = subArray.splice(subArray.length-1).join('/');
}
```
If we're in a subfolder, we need the ability to navigate to the parent folder. The above is prep work for that UI element. If the `subPath` variable is not an empty string, then we create a temporary array by splitting `subPath` with the forward slash as a delimiter. If that array has more than one element, we `splice` off the last element and then rejoin them using forward slashes, to create the new value of the `backPath` variable.
```js
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('name', 'back');
i.setAttribute('value', '⬅️');
i.setAttribute('id', '..');
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = 'Go back';
f.appendChild(p);
$('#folder_list').append(f);
```
The above looks like a lot because it's so many lines of code, but really we're just creating and customizing HTML/DOM elements and strapping them on the existing web page. We make a `form`, with an `input type="button"` inside, and a `p` below that.
```js
document.getElementById('..').addEventListener('click', (e) => {
console.log(e.target.id);
$('#folderNameDisplay').html(
(backPath === '') ? 'Root folder' : backPath
);
$('#subPath').val(backPath);
```
This is our click listener for the back button. We conditionally set the `innerHTML` of the element with ID `folderNameDisplay` to either the backPath (if it's not an empty string), or the words 'Root folder' if `backPath` was an empty string. We also set the value of an `input type="hidden"` element with ID `subPath` where we keep track of the currently-viewed folder. Now comes the actual AJAX call:
```js
$.post(
'/media-manager',
{
cmd: 'showDir',
subPath: $('#subPath').val(),
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
}).fail(() => {
})
.always(() => {
});
e.preventDefault();
});
```
The above executes our AJAX call to show us the updated contents of the folder. The first argument to `$.post()` is the URL, the second argument is the data we need to send to that URL, the third argument is a success function, and the fourth argument notifies jQuery that we expect the response to be JSON data. After that I left blanks for error handling (the `.fail()` trigger) and an after-process trigger that will always happen (the `.always()` trigger). Finally: because this is AJAX; because the back button is a submit button; and because we don't want the page to refresh when you click it, we use `e.preventDefault()` to prevent the form from submitting the old-fashioned way.
```js
}
```
We're done with the `if` statement, `if (subPath !== '')`.
```js
folders.forEach((item) => {
```
It's time to loop through the `folders` array and render each folder. If you paid attention above, most of the below should look familiar.
```js
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('name', item);
i.setAttribute('value', '📁');
i.setAttribute('id', item);
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = item;
f.appendChild(p);
$('#folder_list').append(f);
document.getElementById(item).addEventListener('click', (e) => {
console.log(e.target.id);
let val = (subPath !== '')
? `${subPath}/${e.target.id}`
: e.target.id;
$('#folderNameDisplay').html(val);
$('#subPath').val(val);
$.post(
'/media-manager',
{
cmd: 'showDir',
subPath: $('#subPath').val(),
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
}).fail(() => {
})
.always(() => {
});
e.preventDefault();
})
});
```
As shown above, the inside of the `folders.forEach` loop is almost exactly the same code as the code for the "back" button. We:
- create some elements, strap them together, and then strap them onto the web page
- create an event listener for clicking on the submit button which represents the folder
- do the AJAX call in that event listener, and prevent traditional form submission from occuring
Things will, once again, be very familiar when we loop the `files` array, below, but with a slight twist:
```js
files.forEach((item) => {
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('value', '');
let sub = ($('#subPath').val()) ? `${$('#subPath').val()}/` : '';
i.setAttribute('id', item);
```
Here comes the twist:
```js
let url = `/static/img/mm/${sub}${item}`;
i.style.backgroundImage = `url('${url}')`;
i.style.backgroundSize = 'cover';
```
Instead of an emoji of a folder icon, we're using the actual image itself as the CSS `background-image` attribute, represented in DOM as the `.style.backgroundImage` property; and setting the CSS `background-size` attribute to `cover` via the DOM attribute `.style.backgroundSize`. With very few differences, everything else is the same until you get to the click handler.
```js
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = item;
f.appendChild(p);
$('#folder_list').append(f);
```
The click hander starts the same, but...
```js
document.getElementById(item).addEventListener('click', (e) => {
console.log(e.target.id);
```
...then comes this part:
```js
navigator.clipboard.writeText(
``
);
$('#media_manager').fadeOut(150);
$('.copy_msg').fadeIn(1000).fadeOut(5000);
```
Above, we're copying a string to the user's clipboard. This is the same as if they had selected that text on a page and pressed Ctrl-C. We also fade out the media manager, fade in a message, and fade out that message again, slowly. (The message explains that they can now paste the Markdown code.)
```js
e.preventDefault();
})
});
}
```
We prevent the form from submitting the non-AJAX way, end the click handler, end the `files.forEach()`, and end the function.
Now we do our page setup. The following code runs as soon as the browser is ready to render the page:
```js
$(document).ready(async () => {
// Media manager always starts at the root folder
$('#folderNameDisplay').html('Root folder');
```
Above, we set an H3 above the folder viewer to say "Root folder" because that's where the media manager starts.
```js
// initialize jquery-uploadfile plugin
$("#fileuploader").uploadFile({
url: "/media-manager",
fileName: 'uploadedFile',
multiple: false,
dragDrop: false,
dynamicFormData: () => {
return { cmd: $('#cmd').val(), subPath: $('#subPath').val(), };
},
returnType: 'json',
onSuccess: (files,data,xhr,pd) => {
console.log(JSON.stringify(data));
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
},
});
```
The above code initializes the jQuery-uploadfile plugin. We use the `dynamicFormData` field to allow us to feed the AJAX call up-to-date data from our `subPath` field. The `$.post()` call in its `onSuccess` field should be familiar by now.
```js
// setup create folder button
document.getElementById('create_folder')
.addEventListener('submit', (e) => {
```
Now we're going to add an event listener for the `submit` event on the `create_folder` form.
```js
let newFolder = $('#new_folder_name').val();
```
We grab the value of the text input box where the user types the name of the folder they want to create.
```js
if (
newFolder === ''
|| newFolder.indexOf('/') > -1
|| newFolder.indexOf('\\') > -1
) return false;
```
We do some basic security: no slashes allowed.
```js
$.post(
`/media-manager`,
{
cmd: 'mkDir', subPath: $('#subPath').val(),
newFolder: newFolder,
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
console.log(data);
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
```
First we send the `mkDir` value in the `cmd` field of the POST request, along with the `subPath` and `newFolder` fields. Then, when that request reports done, we refresh the folder view.
```js
.fail(() => {
})
.always(() => {
});
})
.fail(() => {
})
.always(() => {
});
```
As before, the above are just blanks you can fill out if you want to.
```js
e.preventDefault();
```
Prevent the form from submitting the non-AJAX way.
```js
});
```
Close out the click handler.
Next we populate the media manager with a view of the root folder (that is, the folder it's configured to treat as its `basePath`). It's the same as the other "get data/render the folder list" chains.
```js
// get initial folder data
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
```
Next we set up the close and open buttons to do their things:
```js
$('#close_x').click(() => {
$('#media_manager').fadeOut(150);
});
$('.open_mm').click(() => {
$('#media_manager').fadeIn(150);
});
```
And we're done with the script that runs when the page is ready for display!
```js
});
```
### Reminders
Do not expose this to random users. It's not secure enough. I wrote this in a morning from scratch. Make sure you only allow competent admins to use this! But with that said, I think it's attractive, fast, and gets the job done.
Featured Article
How to Code Relevance-Weighted Search with Node & MongoDB
By AstroMacGuffin dated Sat Jul 23 2022 19:14:09 GMT+0000 (Coordinated Universal Time) last updated Thu Aug 18 2022 14:17:35 GMT+0000 (Coordinated Universal Time) In this post I teach you how to implement a search that sorts results by relevance. Search relevance is calculated as the total number of times each search keyword shows up in the article, added together. To accomplish this, you'll create a search index whenever the article is saved.
Regular expressions will be used to match the keywords against the mongo database, so the search keyword "search" will match the database keyword "searching", but the search keyword "searching" won't match the database keyword "search".
### Ingredients
- some kind of CRUD app already written for articles using Node & MongoDB
- an `articles_search` collection for storing strength of keyword relevance per word per article
- a method for removing punctuation and special characters from the content you want to index
- database methods for deleting and inserting into `articles_search`
- a mongo query that matches search terms against a field in `articles_search`
- a loop to add up total keyword strength per article
- a sorting algorithm to sort articles by total keyword strength
Whenever you add or update an article:
- delete all keyword entries for that article
- parse the contents you want to index from scratch; delete old entries for that article from `articles_search`, then add new
- update the article content normally
For the search, it's actually two find operations (probably because I haven't looked into how MongoDB handles join operations yet). First you get the entries from `articles_search` so you can have the `articles._ids`; second you get the article content.
Some of my code to get you through the relatively hard parts:
### First, a Useful Utility Method
```js
_renderSearchTerms(theString) {
return theString
.replace(/\(/gi, ' ')
.replace(/\)/gi, ' ')
.replace(/\./gi, ' ')
.replace(/!/gi, ' ')
.replace(/@/gi, ' ')
.replace(/#/gi, ' ')
.replace(/\$/gi, ' ')
.replace(/\%/gi, ' ')
.replace(/\^/gi, ' ')
.replace(/&/gi, ' ')
.replace(/\*/gi, ' ')
.replace(/-/gi, ' ')
.replace(/_/gi, ' ')
.replace(/=/gi, ' ')
.replace(/\+/gi, ' ')
.replace(/\{/gi, ' ')
.replace(/\[/gi, ' ')
.replace(/\}/gi, ' ')
.replace(/\]/gi, ' ')
.replace(/:/gi, ' ')
.replace(/;/gi, ' ')
.replace(/"/gi, ' ')
.replace(/'/gi, ' ')
.replace(/`/gi, ' ')
.replace(/,/gi, ' ')
.replace(/>/gi, ' ')
.replace(/</gi, ' ')
.replace(/\//gi, ' ')
.replace(/\?/gi, ' ')
.replace(/\|/gi, ' ')
.replace(/\\/gi, ' ')
.replace(/\n/gi, ' ')
.split(' ');
}
```
The above utility method:
- removes all the unwanted characters from a string you intend to index for search, replacing them with spaces
- splits the result into an array, using a space as the split delimeter
- returns it
### Whenever you Add or Update an Article, Set the Keywords Info in `articles_search`
Next is the method that sets the `articles_search` entries for an article.
```js
async setSearchTerms(arg) {
const d = mdb.db("amgdotcom");
const t = d.collection("articles_search");
let id = (arg._id !== undefined) ? arg._id : arg.id;
if (id === undefined) return false;
await this.deleteSearchTerms(id);
```
We don't want duplicate entries, so we've deleted all the old search terms for this article.
```js
let terms = [
...this._renderSearchTerms(arg.title),
...this._renderSearchTerms(arg.content)
];
```
This creates a merged array; note that there may very well be words in the `title` that are also in the `content` ...that's fine, because right now the `terms` array contains the entire article as an array broken into words -- no sums yet, just a raw list of words exactly as they appeared in the original article. Not until we...
```js
let termCount = {};
terms.forEach((item) => {
item = item.trim().toLowerCase();
if (item !== '') {
if (termCount[item] === undefined) termCount[item] = 1;
else termCount[item]++;
}
});
```
*Now* we have the sums. Next we simply collate a `docs` array full of objects compatible with MongoDB's node driver method, `[db].[collection].insertMany()`.
```js
let docs = [];
// rewriting terms var
terms = Object.keys(termCount);
terms.forEach((item) => {
docs.push({
article_id: ObjectId(id),
term: item,
count: termCount[item]
});
});
```
Finally we insert the `articles_search` entries into the collection (`t` was defined all the way on the 2nd line of this function).
```js
t.insertMany(docs);
}
```
### And Finally, the Search Method
```js
async searchArticles(term, type='blog') {
```
The `term` parameter needs to be a space-separated string of keywords.
```js
try {
let r = [];
let article_weights = {};
const d = mdb.db("amgdotcom");
let t = d.collection("articles_search");
```
Just setting up variables to be used later. The collection `t` will be reassigned later, so it's not a `const` like it otherwise would be.
```js
let query = { $or: [] };
let terms = term.split(' ');
for (let i = 0; i < terms.length; i++) {
query['$or'].push({term: new RegExp(terms[i], 'gi')});
}
```
We're dynamically building a `query` object, above.
```js
let options = {
sort: { count: -1 },
projection: { article_id: 1, count: 1 },
};
```
All we need from the `articles_search` collection is the `articles._id` value stored in `articles_search.article_id`, and the `count` of how relevant the keyword was to that article. We don't care about the keyword, but we do sort by `count`, descending.
```js
const cursor1 = await t.find(query, options);
await cursor1.forEach((item) => {
if (!article_weights.hasOwnProperty(item.article_id))
article_weights[item.article_id] = { count: item.count };
else
article_weights[item.article_id].count += item.count;
});
```
Above, we're priming the `article_weights` object now, calculating the total relevance. The goal is to know how relevant the article is when you take all keywords in the search term into account at once.
```js
// resetting query
if (type === 'all')
query = { $or: [] };
else
query = { type: type, $or: [] };
let article_ids = Object.keys(article_weights);
for (let i = 0; i < article_ids.length; i++) {
query['$or'].push({_id: ObjectId(article_ids[i])});
}
```
Once again, we're building a dynamic query above. This time we want every article that's relevant at all.
```js
// resetting t
t = d.collection('articles');
// resetting options
options = {
projection: {
_id: 1, title: 1, content: 1, tags: 1, username: 1,
creation: 1, updated: 1, type: 1,
},
};
```
Now we're preparing to do a `.find()` on the `articles` collection. I didn't bother sorting on the lazy assumption that I have to do that manually. (Seriously, I should just look up how MongoDB joins work, if at all.)
```js
const cursor2 = t.find(query, options);
```
I love the MongoDB Node.js driver so far.
```js
await cursor2.forEach((item) => {
r.push({
id: item._id,
title: item.title,
content: item.content,
tags: item.tags,
username: item.username,
creation: item.creation,
updated: item.updated,
type: item.type,
});
});
```
The `r` array is the return value. Here we're packing it full of everything needed to render each article that was relevant to the search.
```js
r.sort((a, b) => {
if (article_weights[a.id].count > article_weights[b.id].count)
return -1;
else if (article_weights[a.id].count === article_weights[b.id].count)
return 0;
else return 1;
});
return r;
```
Above, we sort the return value so that the more total keyword matches for an article, the earlier in the array it'll be sorted.
```js
}
catch (e) {
console.log(`There was an error: ${e}`);
}
}
```
The other methods are easy, basic CRUD stuff. Have fun with your fancy new relevance-weighted search!
