Ref Home!
Series: "How to Code: The Fundamentals"
By AstroMacGuffin dated Wed Sep 14 2022 10:54:19 GMT+0000 (Coordinated Universal Time) last updated Wed Sep 14 2022 10:54:24 GMT+0000 (Coordinated Universal Time)This is just an easier way to find the articles, in order:
- [Lesson 0: Orientation & Absolute Basics](https://astromacguffin.com/ref/id/62ee85c9ae620ee833fd88b7)
- [Lesson 1: Control Flow, Scope, Variables, Types, & Functions](https://astromacguffin.com/ref/id/62ebac50209885383e63d856)
- [Lesson 2: Async/await, Objects & Classes](https://astromacguffin.com/ref/id/62ec1c6a209885383e63d857)
- [Lesson 3: Data Structures & Loops](https://astromacguffin.com/ref/id/62f27794719797d9d45be421)
- [Lesson 4: Checkpoint Project: Logic & Workflow Basics](https://astromacguffin.com/ref/id/6309fb281b7ad5122a6e6aea)
More to come!
Featured Article
How to Code: The Fundamentals, pt 4
By AstroMacGuffin dated Sat Aug 27 2022 11:08:24 GMT+0000 (Coordinated Universal Time) last updated Thu Sep 08 2022 11:43:14 GMT+0000 (Coordinated Universal Time)There's one major difficulty in writing this series: everything should come first. That is to say, whatever I could possibly put into lessons 2, 3, or 4, it should have been in lesson 1. The goal of this series is to write for people who are frustrated because project-based tutorials can only teach so much. You've slammed into some bugs and don't have the fundamental JavaScript knowledge to fix them. But specifically JavaScript-related knowledge isn't the end-all, be-all of coding in JavaScript. To write good code in any language, you need to grok **logic**.
But what does that mean? For programmers, logic means breaking your tasks down to tiny steps: finding where the `if` and `else` statements should go, along with the other control flow features; specifying exactly what those `if` statements will hinge on; figuring out what variables and functions you need; discovering what those functions should take as input (parameters), and give as output (return values); checking the specifics of any API's you plan to use, so that you can account for their input and output formats; and so forth.
When I was a kid, we did all this and turned it into flowcharts, but I think **pseudocode** is better. Starting with pseudocode puts you in the trenches of your project and allows you to seamlessly switch between writing code and pseudocode, inline with each other. But what's perhaps even better than pseudocode, for a beginner doing beginner projects: a plain old list.
Mastering programming logic leads to making fewer mistakes. Each mistake can easily cost you an hour or more. Let's get started.
### A Casual Approach to Logic
We won't be throwing around any fancy logic terms in this lesson. You already know logic: you just need to learn how to apply what you know. Remember, the biggest step in preparing to write code is breaking the task down into tiny bits. You've probably made a list of steps for some kind of procedure before - this is no different. In fact, before (or instead of) jumping in to pseudocode, you may find it helpful to make a simple checklist. If you do, make sure you leave plenty of vertical space between each entry on the list, so you can squeeze in more steps as you realize they're needed.
### Blasting Off Again!
Let's give you a very simple task: make an ASCII-art rocket ship blast off on an HTML page. To make the ASCII-art look like it should, we'll be using the HTML <`pre`> tag, which renders the text within using a fixed-width font. First, you need a framework for the HTML page:
```html
<!doctype html>
<html>
<head>
<title>Time to Blast Off!</title>
<style>
body { background-color: #202020; color: #dcdcdc; margin: 0; }
#launchpad { width: 100vw; height: 100vh; margin: 0; position: relative; }
div { width: 100vw; height: 100vh; margin: 0; overflow: hidden; }
</style>
<script>
document.addEventListener('DOMContentLoaded', () => {
// most of your JavaScript code will go here
} );
</script>
</head>
<body>
<div>
<pre id='launchpad'></pre>
</div>
</body>
</html>
```
You'll notice there is already a non-empty <`script`> tag in the above HTML. `document.addEventListener()` is a standard JavaScript feature that tells the browser to run a callback argument whenever the named event occurs. The callback is empty in the above template; you'll fill it in as we go. The `DOMContentLoaded` event fires when the HTML is ready to be displayed.
There's also a <`pre`> tag with the `id` **launchpad**; this is where the animation will occur. And there's a little bit of <`style`> information, to give you a dark page and light text, and to make the <`pre`> tag take up the whole browser. With these things all pre-filled, we are free to focus on developing the guts of the callback argument, which is where you'll put your JavaScript code.
#### Assembling Your Rocket
Next we need an ASCII-art rocket. I found one with a quick web search, stripped the decals off, and prepped it for launch. (Thank you to [ASCII.co.uk](https://ascii.co.uk).) You're welcome to do your own search and find one you like better, or just draw one in your favorite plain text editor.
What do I mean by "prepped it for launch?" Firstly, I turned it into an array, which means each line is quoted, and has a comma after it, and the whole thing is wrapped in square brackets. Secondly, rockets are aerodynamic things, and so there are slashes and backslashes. Backslashes are special characters in a JavaScript string; in order to say "no really, I actually just want a backslash", you have to do two of them back-to-back: `\\`; this is called "escaping" the character. There were also apostrophes in the rocket; since I used apostrophes as my quote character, those also needed to be escaped with a backslash: `\'`. This is why the rocket looks a little bit shaggy in the code. The following can go either in the callback, or above it:
```js
const rocket = [
' /\\',
' //\\\\',
' ||##||',
' //##mm\\\\',
' //##*mmm\\\\',
' //###**mmm\\\\',
' //###***nmmm\\\\',
' //####***@nmmm\\\\',
' ||####***@nnmm||',
' ||####**@@@nnm||',
' |______________|',
' | |',
' \\____________/',
' | |',
' /| /\\ |\\',
' /_| || |_\\',
' | |',
' | |',
' | |',
' /| /\\ |\\',
' / | || | \\',
' / | || | \\',
' / /\\ || /\\ \\',
' |__/ \\ || / \\__|',
' /____\\ /____\\',
' | | | |',
' | |______| |',
' | | /--\\ | |',
' |____|/----\\|____|',
' \\||/ //##\\\\ \\||/',
' /##\\//####\\\\/##\\',
' //##\\\\/####\\//##\\\\',
' ||/::\\||/##\\||/::\\||',
' \\\\\\\'\'///:**:\\\\\\\'\'///',
' \\\\\\///\\::::/\\\\\\///',
' \\\\//\\\\\\::///\\\\//',
' \\/\\\\\\\\..////\\/',
' \\\\\\\\////',
' \\\\\\///',
' \\\\//',
' \\/',
];
```
If you want to see what the rocket looks like without all the escaping, you can put the following code into the callback, save the file as `rocket.html`, open that file in your browser, and then open the Developer Tools and click the Console tab at the top:
```js
console.log(rocket.join('\n'));
```
We coded the rocket as an Array, so we use `rocket.join()` to convert it into a string, with newline characters (`\n`) between the elements.
#### From Here to There
Now, we haven't covered DOM yet in the previous lessons, and we'll be using DOM features to accomplish the blast-off. So how do I expect you to think for yourself when you don't know what to do? This is where a crucial programming skill comes in: searching for solutions to individual problems. For example, create a new tab in your browser and search for "javascript convert array to string" and you'll find the `.join()` method. If you want even better results, try searching for "mdn convert array to string" -- this will put the Mozilla Developer Network links at the top. (MDN is the definitive source for information about JavaScript.)
Now, with that in mind, create a list of steps you think will go into solving this task. Some crucial information to serve as hints:
- You have a <`pre`> inside of a <`div`>.
- The <`pre`> is already set for `relative` positioning.
- The <`div`> is set to hide anything inside it that travels or extends outside the bounds of the <`div`>.
Don't worry about how you'll accomplish the steps. Just break it down into logical steps.
After you think you've got it, compare your list to mine:
1. Move the <`pre`> down so that it's too far down to be drawn (hidden by the <`div`> and its `overflow: hidden` style rule).
2. "Draw" the rocket in the <`pre`> (put the text inside the element).
3. Move the <`pre`> gradually up, until it's too far up to be drawn (again, hidden by the <`div`>'s `overflow: hidden` style rule).
If you went into more detail, that's great. But for those of you whose list looks more or less like the above, we'll avoid spoilers and take things one step at a time.
Now take another crack at your list and try to imagine what your tasks translate to in JavaScript terms. I don't expect you to know the actual JavaScript features you'll use: I just want you to rephrase your list in more technical terms. In fact, what I really want is for you to rephrase your list in terms of web search queries.
Go ahead, and then compare your list to mine:
1. Change HTML element position
2. Change HTML element contents
3. Change HTML element position
When you think in those terms, your 3-item list just became a 2-item list plus a repeat. That makes things simpler. Now add the word "JavaScript" to your search terms and actually do the first search.
### Step 1: Positioning the Launchpad
#### Overflowing with Knowledge
In case you've never heard of it, there's a website called StackOverflow that is easily just as valuable as MDN. StackOverflow is the internet's best Q'n'A website for nerds. When you do a search and get back links to StackOverflow with titles very similar to your search query, that usually means you're one click away from your answer. When I search for "javascript Change HTML element position", the first two results are perfect matches:
The first link has the correct answer: to move an element up and down, you get a reference to the element, and then set its `.style.top` attribute. Easy peasy, right? Except, the example code at StackOverflow has them setting the `.style.top` as a number of pixels, e.g. `123px`. That's not what we want. We don't want to figure out how many pixels tall the browser is; and we don't want our code to break on different-sized browsers. So, we're going back to the search engine for another option besides the `px` unit.
This one's a little tricky. How do you phrase the search query? We want to know what our options are when it comes to units for positioning an HTML element. But when I search for "units for positioning an HTML element" I get nothing useful. What about another approach: I'll search for "mdn alternatives to px". For that, the first result gives me an article about `length` as a CSS data type. These are CSS properties we're setting in the `.style` field, so we click the article and start reading. If you do that, you should quickly get the hint that this is where we need to be. About a third of the way down the page, we get our solution.
Instead of `px` we'll use the units `vw` and `vh`, which are percentages of the viewport width and height, respectively. That is, `1vw` equals 1% of the viewport width, and `1vh` equals 1% of the viewport height.
There's one more detail we've missed. We learned how to set the positioning of an element, but it required our code to have a reference to that element in DOM. Let's try the search "javascript get reference to element". My first search result is another StackOverflow page which gives me the knowledge that we'll be using `document.querySelector()`. But the example seems much too specific and doesn't make it clear how we'd select our <`pre`> tag. Armed with a bit more knowledge, we go back to the search engine and query for "mdn document.querySelector". This tells us that the sole parameter to that method is a `selector` which must be a valid CSS selector string. We're almost there. Search for "CSS select by ID". This time my first result is from W3Schools, which is another quality website. It's specific, and it gets to the point: the selector for a tag with `id="whatever"` is `#whatever`.
#### Committing it to Code
So now we know how to accomplish step 1 on the breakout list. We've learned:
1. We use `document.querySelector()` to get a reference to an element.
2. The selector should be `#launchpad` because that's the `id` of the <`pre`> tag.
3. Once we have that element, we set `.style.top` to change the up-and-down position of the element.
4. To move the element all the way down the page, we probably set `.style.top` to `100vh` or 100% of the viewport's height.
Now we commit what we've learned to code, and try it out. I suggest you try to write the code yourself and test it, before looking at the solution below:
```js
// capture the pre element
const el = document.querySelector('#launchpad');
// move the element below the viewport
el.style.top = '100vh';
```
Add it to your callback code, save it, and reload the browser. Open the Developer Tools and make sure you're on its "Elements" tab. Navigate the document structure until you can see the <`pre`> tag in the source. Hover your mouse over the source code for the <`pre`> tag. You will see a bubble appear in the browser's viewport, showing you that the <`pre`> is indeed below the viewport. And we don't have a scrollbar, so the element really is hidden!
Coder-space, here we come!
### Step 2: Inner Space
Rinse and repeat. Go back to the second breakout list, where we rephrased things into search terms. Copy-paste step 2 into a search engine and add "JavaScript" to the search. For me, the top result is another W3Schools page, and it has the answer: you set the element's `.innerHTML` field to replace its contents. And since I've already shown you how to convert the `rocket` Array into a string, this step is pretty much one-and-done. I'll show you again, so you don't have to scroll around and lose your place:
```js
console.log(rocket.join('\n'));
```
The `console.log()` method, of course, is how we write to the Developer Tools console; now try adapting the above into a solution for step 2 and get that rocket painted onto the <`pre`> tag. Try it yourself, and then compare to the correct solution below:
```js
// draw rocket on pre
el.innerHTML = rocket.join('\n');
```
Some of you will have done this instead:
```js
// incorrect
el.innerHTML(rocket.join('\n'));
```
...but `.innerHTML` is a field, not a method.
Let's do a reality check and see that it worked. Save, reload, and go to the Developer Tools. On the Elements tab, if you expand everything so that the <`pre`> tag's contents are shown, you'll see... a mess. The Developer Tools trims out a lot of the whitespace, so we'll just have to wait before we see whether the rocket is indeed being rendered in its full glory. We'll get there... by blasting off.
### Step 3: Countdown
Usually the countdown occurs *before* the rocket blasts off, but in our case, a countdown is *how* our rocket will blast off.
Step 3 is to *gradually* change the <`pre`> tag's position until it's up, up, and away. But how do you do things *gradually* in JavaScript? When we set `el.style.top = '100vh'` it teleported there. If we set it again for this step, won't it just teleport above the viewport, with no visible steps?
That depends on what you set `el.style.top` ***to.***
Think about it. We're doing animation. Our <`pre`> tag currently has a `top` position of `100vh`, which is what moved it 100% of the way down. The next step, logically, would be to set its `top` position to `99vh` or perhaps even `99.9vh`. To do things gradually, in JavaScript and every other language out there, we do them *a step at a time.*
Let's do this animation as smoothly as we can, and decrease the `top` field by `0.1vh` at each step. But the question remains: how do we do this in steps? I'm definitely not asking you to write a hundred, or a thousand lines of code. (Spoiler: It would actually be 2,000 lines of code, if you did it that way, but I'll explain why later.) In the previous lesson, we learned about `for` loops. Each time any loop repeats, that's equivalent to doing a step, right? The traditional `for` loop even (usually) hinges around a variable that goes up or down until it hits some limit! Yes, the `for` loop is our solution.
In the previous lesson, I demonstrated the `for` loop as follows:
```js
let something = ["truck", "train", "aeroplane"];
for (let i = 0; i < something.length; i++) {
console.log( something[i] );
}
```
That is, we were working with Arrays as both the limiter of the `for` loop, and as the subject for the loop's "payload" code. You might think we're doing that again because the `rocket` variable is an Array, but no. We just need a number to start at `100` and work its way down, `0.1` at a time.
But how far down should the number go? To `0`? If we set `el.style.top` to `0`, what do you think would happen? The top of the <`pre`> element would be `0vh` from the top of the viewport. That's not a blastoff. It's a start, but we need to go further. How much further? For a true blastoff, we need the *bottom* of the <`pre`> to be `0vh` from the top of the viewport.
Take a look at the <`style`> section above the JavaScript. The <`pre`> tag has a `height` of `100vh`. That means, if we want the *bottom* of the <`pre`> tag to be `0vh` below the top of the viewport, we need to go another `100vh`... we need a countdown that goes from `100` to `-100`.
We're 2/3rds of the way to figuring out the `for` loop itself. But in my example code above, we used the Increment Operator on the `i` variable to increase our iterator and guarantee we aren't creating an infinite loop. In the example above, `i` starts at `0` and goes up by `1` until it's equal to or greater than `something.length`. We don't want it to go up or down by a whole number, so we can't use `++` or `--`. But maybe there's an operator that does something slightly different?
Operators. There's gotta be a list, right? Try searching for "mdn javascript operators" and you'll find the answer. But wow, that's a big page. What were we trying to do? Subtract. How about we speed things along... do a find-in-page (Ctrl-F on PC) and type in "subtract". The first two results are in the sidebar: "Subtraction" and "Subtraction assignment". We can assume the "Subtraction operator", the minus sign, does what we expect it to do... but what is the Subtraction assignment operator? So we click it. The technical explanation might not be written in the clearest language, but the example code is plenty clear:
```js
let a = 2;
console.log(a -= 3);
// expected output: -1
```
This looks perfect. Instead of `i++` our iterator will be modified with the `-=` operator after each run through the loop, using `i -= 0.1`.
Go ahead and take a crack at writing the structure of the `for` loop without any code inside the curly braces. When you're done, compare it to the correct answer, below:
```js
for (let i = 100; i >= -100; i -= 0.1) {
}
```
Some of you might have done this:
```js
for (let i = 100vh; i >= -100vh; i -= 0.1) {
}
```
...but `100vh` isn't a number in JavaScript -- you'd get an error trying to run that code. And if you set `i` to `'100vh'` as a string, things would be badly broken as well. We just need the number for now -- we'll add the `vh` part in the body of the loop.
JavaScript does a thing called "type coercion" whenever you ask it to add, for example, a string and a number. It converts the number to a string and puts the two strings together. That's how you'll add the `vh` to our `i` iterator. Go ahead and take a crack at writing the one line of code that goes inside the `for` loop's body. Compare your results to the answer below:
```js
for (let i = 100; i >= -100; i -= 0.1) {
el.style.top = i + 'vh';
}
```
Don't get your hopes up... programming is a process of steps, just as surely as that loop is a process of steps.
Add it to your callback. Save it. Reload the browser.
Wait... what's wrong? You didn't see anything? Check the position of the <`pre`> tag again, by hovering over it in the Elements tab of the Developer Tools. Well, as expected, it's now above the viewport. It moved too quickly to be seen! You can confirm this by changing the `for` loop to end at `0` instead of `-100`.
So what do we do? We need a delay on these steps. Back to the search engine: "how to delay javascript code". The 2nd result is from StackOverflow and it contains the answer: `setTimeout()`. Now we search for "mdn setTimeout" for details on how it works. It's pretty simple: for input, it just needs a function to run, and a number of milliseconds to delay before running that function.
Spoiler alert: we're really, truly, almost done. Your ride to the stars awaits...
Two things are hopefully obvious at this stage:
1. The function you provide to `setTimeout()` needs only contain one line of code, the line that sets `el.style.top`.
2. The real problem here is the delay. What do we base it on?
Try and work that out yourself before reading on. Remember, a millisecond is an extremely short period of time.
#### Count-ups within Countdowns
Here's the reason the `delay` parameter is a puzzle: if you give `setTimeout()` the same `delay` argument each time through the loop, it's going to animate just as quickly as it did before, and you're not going to see anything. The only difference will be a short delay at the beginning before the `setTimeout()` callbacks start firing, one after the other, at the exact same speed as when you weren't using `setTimeout()` at all. You need to queue up the animation frames. That is to say, they need to be staggered in time; for example, they need to happen 5 milliseconds apart.
Do you think you figured it out? Give it your best try before reading on.
Here's a couple of hints: the Multiplication operator is `*`. And, here's how callback arguments are usually written, using `setTimeout()` as an example:
```js
let delay = 100;
setTimeout( () => {
// callback function body
}, delay);
// the callback is the part that looks like:
() => {
// callback function body
}
```
Do you give up? Here's the answer:
```js
let x = 1;
for (let i = 100; i >= -100; i -= 0.1) {
setTimeout( () => {
el.style.top = i + 'vh';
}, 5*x);
x++;
}
```
We needed a new variable `x` because `i` had an unsuitable number range, going into the negatives. That new variable needed to increase by one, in each step of the loop. Then the "animation frames" could be staggered apart in time: each step of the loop, we multiply `x*5` for the number of milliseconds to delay. The first frame happens at 5ms, the second frame at 10ms, etc. (You can play with that `5` to try faster and slower blastoffs.)
That's it! Save the updated JavaScript file, reload the browser, and watch your rocket fly away!
### Victory Lap: A Pseudocode Approach
What is pseudocode? It's just code comments. It's your breakout list, fleshed out even further, and turned into programming documentation. Because we write pseudocode in the actual code editor, we can write code along with it, at the same time. The best pseudocode is indented the way you expect the code to be indented.
To make sense of that, here's the completed project, as pseudocode:
```
// create rocket variable as array
// event handler: when document is ready
// reality check: output rocket to console
// capture the pre element
// move the pre below the viewport
// draw rocket on pre
// initialize delay multiplier x at 1
// loop i from 100 to -100 step -0.1
// do actual move on a delay
// do move
// stagger the moves x*5ms
// increment x
```
### Conclusion
As this series evolves, I'll periodically help you check your progress. I'll do that via a logic lesson, containing a complete mini-project, using some of the things you've learned in the preceding lessons. It's important to make projects and gain real world experience, but it's arguably more important to understand what you're doing, while you're doing it.
Featured Article
How to Integrate Your Node/Express Website with Google Authenticator for Two-Factor Authentication in 2022
By AstroMacGuffin dated Sat Aug 20 2022 08:55:37 GMT+0000 (Coordinated Universal Time) last updated Mon Aug 22 2022 22:16:57 GMT+0000 (Coordinated Universal Time) I've been making personal websites off-and-on since the 90's, so one can probably assume I've lost websites to hackers in the past. Once the rootkit is installed on your host and the sketchy ads are in your articles, it's very difficult to get them all out. So security is best handled as a preventative measure, not a reactive one. In this article, I'll walk you through the entire process of integrating with Google Authenticator for 2FA (two-factor authentication). I'll be using MongoDB, Node.js, and Express.js.
If you've seen other articles covering this topic, you may have noticed that they use `speakeasy`, an NPM package that hasn't been updated in 7 years. That's not the best look for a security kit. Thankfully, another author has forked that package and updated it just this year, so we're good to go. Other articles also tend not to cover the front-end, which is crucial to understanding how 2FA is even supposed to work. This article takes you through the entire integration.
### Ingredients
- your existing login system
- the NPM package `express-session`, which you should already be using in your login system
- the NPM package `connect-mongo`, probably
- the NPM package `@levminer/speakeasy`, a fork of `speakeasy`
- the NPM package `qrcode`
- a couple of database methods
- a significant rejiggering of the registration and login processes
### Basic Orientation
Maybe this is your first experience with Two-Factor Authentication (2FA). Let's start slow, and explain things as we go:
#### What Is Two-Factor Authentication?
For those of you not up to speed, 2FA means the login has two prompts: one for username and password, and then another prompt for a separate secret. In theory (and practice for companies in the Forbes 500 list) this can mean thumbprints, retinal scans, heck why not even prick a finger and analyze some DNA... in reality, it means whipping out your phone, looking at Google Authenticator, and punching in a 6-digit code. It's a quick and painless process once you get used to it.
#### What are the Advantages of 2FA?
Two-factor authentication protects users and website owners from weak or stolen passwords, by using decently-strong encryption on a physical key. In this case, the key is a smartphone.
#### What is Google Authenticator?
Google Authenticator is a free smartphone app by Google. It retrieves encryption settings from a website by scanning a QR Code. The QR Code is a very important secret, like a password, except that the QR Code must be unique to each user. Once configured, Google Authenticator then issues 6-digit codes, which are entered as a "second password" for logging into 2FA-enabled sites and apps. The codes spawn and expire based on 30-second timers.
### The Nitty Gritty: NPM Packages Used in 2FA
Aside from the usual suspects, i.e. Express and your chosen database connector library, there are three (or four) NPM packages you'll need for this feature.
#### 2FA Validation & URL Generation by `@levminer/speakeasy`
After 7 years of no updates to the `speakeasy` library, Github user `@levminer` has forked the project, reviving and modernizing it. They've also upgraded the `speakeasy` documentation significantly. This library provides the following things that are relevant to our technique for enabling 2FA:
- creation of a shared secret, needed to synchronize encryption/validation between Google Authenticator and your website
- validation of tokens against the shared secret
- creation of an `otpauth` URL for configuring Google Authenticator
`@levminer/speakeasy` also handles *creation* of time-based tokens, useful to those who send the 6-digit code over SMS texts or some other method instead of Google Authenticator; as well as another token system for one-time passwords. But we'll be using only the above three features.
To install:
```
npm install @levminer/speakeasy
```
[Documentation for @levminer/speakeasy](https://www.npmjs.com/package/@levminer/speakeasy)
#### QR Code Generation by `qrcode`
This one takes very little explaining. My favorite part of `qrcode` is that it can output the code in SVG format. My least favorite part is that the method for creating the QR Code, requires a callback to receive the code. This made things slightly tricky in terms of rendering an Express.js template that included the code. You'll see how I solved that, below. The QR Code we generate will encode the URL created by `@levminer/speakeasy`, which tells Google Authenticator how to generate valid 6-digit codes for your website.
To install:
```
npm install qrcode
```
[Documentation for qrcode](https://www.npmjs.com/package/qrcode)
#### Sessions by `express-session`
A "session" is web tech speak for a server-side variable that links to a client-side cookie. The cookie contains only enough info to identify the session. The session can then be used to store whatever data you want to link to that visitor or account. Is the user logged in? Who as? Is that user an admin? Etc. It's very convenient and reasonably secure, especially compared to the alternative of storing all this data in cookies. The `express-session` package is insanely easy to use; just enable the middleware and you instantly have a working session system with variables automatically populated into `req.session`.
To install:
```
npm install express-session
```
[Documentation for express-session](https://www.npmjs.com/package/express-session)
If you need a quick-start for `express-session`, just put this into your main JS file (the one that you run with `node [filename]`, aka the target of your `npm start` script as found in `package.json`):
```js
const session = require('express-session')({
cookie: { sameSite: false },
secret: 'funky ol gorilla',
});
// use sessions
app.use(session);
```
#### Session Data Storage by `connect-mongo`
But where will the session data be stored if the server restarts? By default, that data would be lost and everyone would be logged out of your website. You need database storage for your session data, to solve this problem, and that's where `connect-mongo` comes in. Again, this library is insanely easy to use. You instantiate an object as part of your initialization for `express-session`, and you're done. It automatically creates a `sessions` collection in your database and keeps it up to date as part of the `express-session` task load.
To install:
```
npm install connect-mongo
```
[Documentation for connect-mongo](https://www.npmjs.com/package/connect-mongo)
If you need a quick-start for `connect-mongo`: remember the code I told you to put into your main JS file above? Modify it like so:
```js
const MongoStore = require('connect-mongo');
const session = require('express-session')({
cookie: { sameSite: false },
secret: 'funky ol gorilla',
store: MongoStore.create({
mongoUrl: `YOUR MONGO CONNECTION STRING`,
}),
});
// use sessions
app.use(session);
```
Be sure to change the `mongoUrl` field to your actual MongoDB connection string, which is made of five fields:
- your MongoDB USERNAME
- your MongoDB PASSWORD
- the HOST the MongoDB server runs on (usually 127.0.0.1)
- the PORT the MongoDB server listens on (usually 27017)
- the DATABASE_NAME relevant to this project
You put them together like the following:
```
mongodb://USERNAME:PASSWORD@HOST:PORT/DATABASE_NAME
```
### Now, Where To Begin?
I felt it necessary to start with a quick reality check, to make sure I understood the documentation correctly. So rather than hacking up my existing code, I created a new Node.js script, and started cherry-picking from the documentation for `@levminer/speakeasy` and `qrcode`. Here's the result:
```js
const speakeasy = require('@levminer/speakeasy');
const QRCode = require('qrcode');
const secret = speakeasy.generateSecret({ length: 32 });
const token = speakeasy.totp({
secret: secret.ascii,
encoding: "ascii",
});
console.log(`token: ${token}`);
const tokenValidates = speakeasy.totp.verify({
secret: secret.base32,
encoding: "base32",
token: token,
window: 2,
});
console.log(tokenValidates);
const url = speakeasy.otpauthURL({
secret: secret.ascii, label: "AstroMacGuffin.com", algorithm: "sha512"
});
console.log(url);
QRCode.toString(url, {type: 'svg'}, (err, qr) => {
console.log(qr);
});
```
Let's break that down:
```js
const speakeasy = require('@levminer/speakeasy');
const QRCode = require('qrcode');
```
Above, I'm just including our two rockstar libraries, as already discussed.
```js
const secret = speakeasy.generateSecret({ length: 32 });
```
This `secret` being created above is used to configure the encryption and validation, so you'll see it being used in every step of the process below.
```js
const token = speakeasy.totp({
secret: secret.ascii,
encoding: "ascii",
});
console.log(`token: ${token}`);
```
I needed a 6-digit code to validate, so I generated one. We *won't* be using the above feature, later in the article: but the above was used temporarily as a substitute for Google Authenticator. I used the `.ascii` version of the `secret` above, and the `.base32` version below, just to verify that it works when you mix-and-match; it does.
```js
const tokenValidates = speakeasy.totp.verify({
secret: secret.base32,
encoding: "base32",
token: token,
window: 2,
});
console.log(tokenValidates);
```
Above, we check whether the `token` validates -- the `token` being our 6-digit code, that is. This is how you'll validate the codes from Google Authenticator, too. The `window` option specifies how old the code can be before it's rejected. Each window is 30 seconds long by default, the same length of time Google Authenticator uses. This length of time is configurable, but we're going to leave it alone for compatibility with Google Authenticator.
```js
const url = speakeasy.otpauthURL({
secret: secret.ascii, label: "AstroMacGuffin.com", algorithm: "sha512"
});
console.log(url);
```
Above, we generate a URL for Google Authenticator or any compatible app. The URL starts with the `otpauth` protocol, and includes the label, the ASCII version of the `secret`, and the algorithm name.
Let's pause for a moment. Let's say you're a person who usually browses the web on a mobile device. Along comes a website that wants you to use two-factor authentication ... which means they want you to scan a QR Code. How are you supposed to do that when the QR Code is being displayed on the same device that has your camera and runs the authenticator app? Are you supposed to point another phone at this phone? One phone for Google Authenticator, another phone for browsing? No. Google Authenticator registers itself as the default handler for `otpauth` links, which means webmasters should be using responsive CSS code to show the QR Code on PC and a plain old link to the `otpauth` URL on mobile.
Finally, about that QR Code:
```js
QRCode.toString(url, {type: 'svg'}, (err, qr) => {
console.log(qr);
});
```
As I said, this method requires a callback to receive the QR Code output. In the above, we output the QR Code as SVG to the console. It works beautifully, but we're going to have to work around it a bit in order to incorporate that output into a template. Be patient, we'll get there.
Now that we've verified that the features all work as expected and that we understand how to use them, it's time to start modifying your project.
### Step 1: Your Own Personal 2FA Library
The first step is to convert the "reality check" code, into code assets for your project. I have a `MiscUtils` class that floats around my project as the object `mu`. Everything that doesn't invoke the database, will go there. So first we put the following at the top of the `MiscUtils` class file:
```js
const speakeasy = require('@levminer/speakeasy');
```
And here are the first methods I put into `MiscUtils`:
```js
makeTOTPSecret() {
return speakeasy.generateSecret({ length: 32 });
}
validateTOTP(totp, secret) {
if (!totp || !secret) return false;
return speakeasy.totp.verify({
secret: secret.base32,
encoding: "base32",
token: totp,
window: 2,
});
}
```
(TOTP stands for "Time-based One-Time Password".)
As you can see, this is a copy-paste of my "reality check" code, with very little modification. Thanks to the `window: 2` option and the default time-step of 30 seconds, we give the user 60 seconds to enter the code; that's plenty of time. These two methods are separated because the secret must be generated at user registration, while the token/TOTP must be validated at login.
This, of course, means we must store the secret in the database. Let's get started on the database code, then. I added these methods to `SiteDatabase`, my object class for database operations:
```js
async setTOTPSecret(secret, username) {
try {
const d = mdb.db("yourDatabaseName");
const t = d.collection("yourUsersCollection");
const query = { username };
const doc = { $set: {
totp_secret: {
base32: secret.base32,
ascii: secret.ascii,
hex: secret.hex,
}
}};
await t.updateOne(query, doc, {upsert: false})
}
catch (e) {
logError(`Error setting TOTP secret: ${e}`);
}
}
async getTOTPSecret(username) {
try {
const d = mdb.db("yourDatabaseName");
const t = d.collection("yourUsersCollection");
const query = {username};
let c = await t.findOne(query);
if (!c.totp_secret) {
const secret = mu.makeTOTPSecret();
await this.setTOTPSecret(secret, username);
return secret;
}
return c.totp_secret;
}
catch (e) {
logError(`Error getting TOTP secret: ${e}`);
}
}
```
Let's break that down.
```js
async setTOTPSecret(secret, username) {
try {
```
Here we define the method signature and open a `try` block. The method needs to be `async` because we'll be `await`-ing some database methods inside.
```js
const d = mdb.db("yourDatabaseName");
```
`mdb` is an instance of `MongoClient`, the main workhorse of the `mongodb` NPM package. Replace `yourDatabaseName` with a recipe for cardboard cake, or your database name. Probably the latter.
```js
const t = d.collection("yourUsersCollection");
```
Replace `yourUsersCollection` with the name of your favorite uncle, or your users collection. Probably the latter.
```js
const query = { username };
```
This is our filter; we only want to update one record, the one with a `username` matching the `username` argument sent to this method.
```js
const doc = { $set: {
totp_secret: {
base32: secret.base32,
ascii: secret.ascii,
hex: secret.hex,
}
}};
```
The above is the data we're updating. The above three object fields, correspond to the three encoding styles available for the `secret`.
```js
await t.updateOne(query, doc, {upsert: false});
```
We don't want an `upsert` (which inserts a new record if there's no match for `query`).
```js
}
catch (e) {
logError(`Error setting TOTP secret: ${e}`);
}
}
```
Above we close out the `try` block, issue our `catch` block, and close out the method. (`logError()` is part of my own logger utilities.)
```js
async getTOTPSecret(username) {
try {
```
Above we start the next method: fetching the secret from the database, for a given `username`.
```js
const d = mdb.db("yourDatabaseName");
const t = d.collection("yourUsersCollection");
const query = {username};
```
Same as in the previous method, we get an instance of the database, then an instance of the collection, and then set a `query` to filter our results down to only the record with a matching `username`.
```js
let c = await t.findOne(query);
```
Above, we do the query and capture the result in `c`.
```js
if (!c.totp_secret) {
const secret = mu.makeTOTPSecret();
await this.setTOTPSecret(secret, username);
return secret;
}
```
If the result doesn't have the `totp_secret` field, we must generate one, save it, and return it. That's what we're doing above.
```js
return c.totp_secret;
```
Otherwise, we return the `totp_secret` from the query result.
```js
}
catch (e) {
logError(`Error getting TOTP secret: ${e}`);
}
}
```
And we close out the `try` block, do the `catch` block, and close out the method.
There's more to do in your personal code library, but we'll come back to that when it's time. Wait for it... okay, it's time.
### Step 2: Modifying the Registration Process for Two-Factor Authentication
It's time to show new users the QR Code they'll need for logging in with 2FA. It's crucial that your users understand there is a mandatory action they must do immediately after registration. They must prepare Google Authenticator for 2FA on this website, or else they will not be able to log in, whether that means now or in the future.
There are roughly three techniques for handling the end of the registration process:
1. The user is automatically logged in
2. The user is redirected to the login page
3. The user is put in an onboarding process
It doesn't matter which you prefer, or which your app currently uses. Mine was redirecting to the login page when I started adding this feature. **No matter what, the user should be shown the QR Code as soon as they complete the most basic registration step** in case they do something wacky like clearing their cookies immediately after registration.
If the user gets up and goes to the bathroom and then forgets to come back, any onboarding process can pick up where it left off. But, if the user can't log in because they didn't set up the necessities for two-factor authentication, you probably won't ever see that user again... or, at best, your database will be full of unused users, plus users who had to sign up a second time, with a secondary email address.
As soon as your website's logic says "yeah, this user should be allowed to log in now", that's when you show them the QR Code and give them instructions on setting up and using 2FA to log in.
#### Handling the User
The anatomy of a registration process is:
- validate & sanitize user inputs
- reject if anything is invalid; return user to registration form and report why
- if the inputs are all valid & sane, add the user to the database
- perhaps log the user in
- redirect to a start page OR display a template that begins an onboarding process
Setting up two-factor authentication is basically an onboarding process, so all roads lead there: if you were redirecting the user, you have to stop doing that and create a template that displays and explains the QR Code. If you were already onboarding the user, you need to bump step 1 down to step 2, and insert a new step 1: preparing for two-factor authentication.
For example, after adding the user to the database, my 2FA post-registration step now does this:
```js
const secret = await db.getTOTPSecret(usernameSafe);
await mu.getTOTPQRCodeAndRenderTemplate(
secret,
res,
'user-adduser',
Object.assign({
title: 'User/User Registered! | AstroMacGuffin.com',
message: 'User/User Registered!',
activeLink: undefined,
}, await db.getDefaultRenderOptionsObj(req)),
usernameSafe,
);
```
There's a lot to unpack here, but mainly:
- we've got a `MiscUtils` method that just appeared out of nowhere! It's the briskly-named `.getTOTPQRCodeAndRenderTemplate()`.
- we've got a `SiteDatabase` method I've never mentioned, `.getDefaultRenderOptionsObj()`.
`.getDefaultRenderOptionsObj()` is simply a method that collects a bunch of data and makes it available to my templates. It saves a lot of work because now my routes don't need logic for a slew of variables and database requests.
Remember I said there was going to be a problem with `QRCode.toString()` because it requires a callback to receive the QR Code? I'm not sure how many solutions there were to this problem, but here's the one I chose. First, we add the following to the top of the `MiscUtils` file:
```js
const QRCode = require('qrcode');
```
And then we add the following method:
```js
async getTOTPQRCodeAndRenderTemplate(secret, res, tpl, obj, username) {
try {
const url = speakeasy.otpauthURL({
secret: secret.ascii,
label: `YourWebsiteName.com (${username})`,
algorithm: "sha512"
});
await QRCode.toString(url, {type: 'svg'}, (err, qr) => {
res.render(tpl, Object.assign(obj, {qr, qrURL: url}));
});
}
catch (e) {
logError(`Can't make QR code: ${e}`);
}
}
```
Let's break that down:
```js
async getTOTPQRCodeAndRenderTemplate(secret, res, tpl, obj, username) {
try {
```
This is an asynchronous method with a long name and a lot of parameters, all of them mandatory.
- `secret` is a TOTP secret, generated by `speakeasy` via our `MiscUtils` helper method.
- `res` is the Express response object. We need this to render the template.
- `tpl` is the name of the template to be rendered.
- `obj` is the data to be passed to the template.
- `username` is the already-sanitized and validated username.
```js
const url = speakeasy.otpauthURL({
secret: secret.ascii,
label: `YourWebsiteName.com (${username})`,
algorithm: "sha512"
});
```
Make sure you replace `YourWebsiteName.com` with your nicest pair of socks. Or, you know. Maybe your website name? The `label` option is what will be displayed in Google Authenticator. It should include the `username` for mainly ***your*** personal convenience, as we all have our admin accounts, our test accounts, and possibly more...
```js
await QRCode.toString(url, {type: 'svg'}, (err, qr) => {
res.render(tpl, Object.assign(obj, {qr, qrURL: url}));
});
```
And the solution to our callback problem. What's this do?
- It uses the `res` object passed into the `.getTOTPQRCodeAndRenderTemplate()` method, to render a template.
- Which template? `tpl`, as passed into the `.getTOTPQRCodeAndRenderTemplate()` method.
- The `obj` (template data passed into the method) is merged with new data using `Object.assign()`.
- The new data contains both the QR code (`qr`) and the URL embedded in the picture (`qrURL`).
```js
}
catch (e) {
logError(`Can't make QR code: ${e}`);
}
}
```
End the `try` block, do the `catch` block, and end the method.
Now your post-registration template just needs to be modified. Out with the redirect (if there was one) and in with the QR Code and an explanation of it and of 2FA.
Here's a tip for you Pug users: If you output the SVG QR Code the wrong way, it will display a bunch of SVG code on your output (i.e. it will display the code on your website) rather than actually using the SVG as source. Here's how to do it:
```
div !{qr}
```
And here's another tip for everyone: you can use CSS classes as conditional logic for your responsive code. That's how my post-registration page knows whether to show the QR Code, or just the link:
```css
/* desktop */
.if-mobile {
display: none;
}
.if-not-mobile {
display: block;
}
@media (max-width: 1024px) {
/* mobile */
.if-mobile {
display: block;
}
.if-not-mobile {
display: none;
}
}
```
Here's the relevant part of my Pug template, which should be self-explanatory:
```
h1= message
p Hang tight, we just need to go over 2 things:
h2 Two-Factor Authentication Required
div.if-not-mobile#qr(style="margin: 2.5vw auto; width: 25vw; align: center;") !{qr}
p.
<strong>Action required! Don't leave this page until done!</strong>
It's simple. Download the free Google Authenticator app on your
smartphone or similar device. Launch it. Click the plus-sign button
on the bottom-right, and choose to scan a QR code. Scan the code
above. That's the setup; you'll only do those steps once.
p.if-mobile
strong.
If you don't see a QR code above, click this link instead:
a(href=`${qrURL}`).
You must have Google Authenticator (or a similar app) installed on
this device.
p.
Then, each time you log in to AstroMacGuffin.com, launch
Google Authenticator and enter the 6-digit code.
```
### Step 3: Modifying the Login Process for Two-Factor Authentication
We're almost done! Now all that's left is to prompt for, and verify, the 6-digit code.
It's up to you whether to make a 3-field login form (username/password/2FA) or a 2-step login form (username and password on the first step, 2FA on the second step). I went with two steps.
The anatomy of a login without 2FA:
- sanitize the username and password inputs
- check the username and password against the database
- if either of the two steps above fails, return the user to the login form with feedback
- otherwise, log the user in and set whatever session variables
- redirect the user to some start page
We need to modify that. We'll be bumping the redirect to later, inserting another step before that redirect.
Let's say, for example, that your Express.js routes are set up like this:
- `/user/login` points at the login page
- `/user/dologin` processes the login form once submitted
And your Pug templates, like this:
- `user-login.pug` is the page with the login form.
- `user-dologin.pug` is shown after successful login; it redirects the user to the home page after 3 seconds.
After we're done:
- The Express.js route `/user/login` and Pug template `user-login.pug` will not be modified.
- The route `/user/dologin` will instead serve the new template `user-2faform.pug` after successful username/password login.
- `/user/do2fa` will be a new route which serves `user-dologin.pug` on successful 2FA login.
**But first let's talk about session variables.** The `/user/dologin` route was previously in charge of setting session variables that indicate to the system that the user is logged in. We still need the data from those variables, but we don't want them to be stored with the same configuration as they are in the unmodified form. I recommend simply moving them to a `req.session.login` object -- all the same variables, but without the side-effect of accidentally treating your users as if they're logged in, before they complete two-factor authentication.
`user-2faform.pug` is a whole page template containing a simple `form` with one `input type="text"` plus a submit button. It doesn't get much easier. The `action` of the `form` is, of course, `/user/do2fa`. We'll name the `input` field `twofactor`.
The `/user/do2fa` route is surprisingly simple:
```js
router.post('/do2fa', async (req, res) => {
if (!req.session.login || !req.session.login.username)
return res.redirect('/user/login');
const secret = await db.getTOTPSecret(req.session.login.username);
const valid2fa = mu.validateTOTP(req.body.twofactor, secret);
if (!valid2fa) {
try {
return res.render('user-2faform', Object.assign({
title: '2-Factor Authentication Failed. Try again!',
message: '2-Factor Authentication Failed. Try again!',
activeLink: undefined,
}, await db.getDefaultRenderOptionsObj(req)));
}
catch (e) { logError(e); return; }
}
req.session.username = req.session.login.username;
delete req.session.login;
res.render('user-dologin', Object.assign({
title: 'User/Login Successful!',
message: 'User/Login Successful!',
activeLink: undefined,
}, await db.getDefaultRenderOptionsObj(req)));
});
```
Let's break that down:
```js
router.post('/do2fa', async (req, res) => {
if (!req.session.login || !req.session.login.username)
res.redirect('/user/login');
```
This is a `post` route. If we haven't signified that the user has succeeded at the username/password login yet -- by setting their username in `req.session.login.username` -- then we kick them back to that login form.
```js
const secret = await db.getTOTPSecret(req.session.login.username);
const valid2fa = mu.validateTOTP(req.body.twofactor, secret);
```
Above we fetch the user's TOTP secret from the database, and then validate their input with our wrapper function. My `username` is sanitized before storage in the session; be sure you are sanitizing it at some point before it finds its way to `db.getTOTPSecret()`, since that method puts the `username` in a database query.
```js
if (!valid2fa) {
try {
return res.render('user-2faform', Object.assign({
title: '2-Factor Authentication Failed. Try again!',
message: '2-Factor Authentication Failed. Try again!',
activeLink: undefined,
}, await db.getDefaultRenderOptionsObj(req)));
}
catch (e) { logError(e); return }
}
```
As shown above, if the 2FA 6-digit code wasn't valid, we kick the user back to the `user-2faform.pug` template with a message that they should try again. We `return` the call to `res.render()` (and return in the `catch` block for extra security) so that users with invalid logins, won't experience the code below:
```js
req.session.username = req.session.login.username;
delete req.session.login;
```
*Now* we can log the user in. Remember when I talked about session variables that signify to your system that the user is logged in? We need to move those out of their temporary holding cell and into their normal configuration. And thus, the user is logged in.
```js
res.render('user-dologin', Object.assign({
title: 'User/Login Successful!',
message: 'User/Login Successful!',
activeLink: undefined,
}, await db.getDefaultRenderOptionsObj(req)));
});
```
Now we can serve the template that redirects users to a start page post-login. We do that simply by serving the same template that was originally served for successful username/password logins before 2FA was integrated.
### Conclusion
Two-factor authentication only works if it's done correctly. My first implementation had a single site-wide secret that was shown on the login page. If I hadn't thought about the fact that this broadcasts the only layer of security involved in 2FA to everyone interested, it would have stayed that way. (With thanks to a friend who confirmed my suspicions that I had done it wrong, and shared what they knew.)
On that note, I should point out that I am not terribly well-versed in security matters. The information provided in this article comes as-is with no claims of usability for any purpose. If you follow this tutorial, I am not responsible for any results. Sorry, had to get a little bit of legal-ese out of my system.
Now the only thing left is to decide whether all logins require 2FA, or whether to make it optional? It depends on how disaster-proof your website is. The higher your risk after security has been penetrated, the more security you need. I went with mandatory 2FA for all logins because I don't like disaster recovery at all. I don't like disasters. At all. The minor speed bump of using two-factor authentication to log into my own website is nothing compared to the multi-day, sometimes weeks-long ordeal of recovering from a hack.
How to Code a Singleton in Node.js
By AstroMacGuffin dated Sun Aug 14 2022 11:02:48 GMT+0000 (Coordinated Universal Time) last updated Sun Aug 21 2022 05:57:06 GMT+0000 (Coordinated Universal Time) This will be a short article because it's an easy topic. Node.js caches your modules when you `require()` them, but there's a caveat: if there's a change in the path to the file being required, it gets treated as a separate module, with a separate cache entry. As long as you code around that hiccup, you'll be slinging singletons in no time.
But first, why are singletons even useful? Imagine you have a game engine. Modern games live and die on events, and your code has to register events in such a way as to ensure all the events are in the same registry. If they aren't, then your event chains won't work: event A is supposed to trigger event B, but if events A and B are registered in two separate objects, that chain of events won't happen.
Of course, doing things the normal way, you'd have this at the top of your file:
```js
const gameEventManager = require('./path/GameEventManager');
```
That module naturally ends with this:
```js
module.exports = new GameEventManager();
```
...and then whenever you need to use the `gameEventManager` you'd do this:
```js
gameEventManager.registerEvent( theEvent );
```
We'll be making some slight modifications to get a Node singleton.
Featured Article
How to Code: The Fundamentals, pt 3
By AstroMacGuffin dated Tue Aug 09 2022 15:04:52 GMT+0000 (Coordinated Universal Time) last updated Thu Aug 11 2022 20:50:30 GMT+0000 (Coordinated Universal Time) In part 2 we dealt with `async`/`await`, objects and classes. In part 3, we're getting into Arrays and a little further into objects, plus we're taking another bite of control flow. These unrelated topics are closely linked in actual use.
In this lesson, we'll cover:
- arrays
- how to find your way around objects and arrays
- data type methods & built-in objects
- comparison operators
- `for` loops and the `Array.forEach` method
- the Increment (`++`) and Decrement (`--`) operators
- `while` loops
### Data Type Methods & Built-in Objects
On a certain level, everything in JavaScript is an object. Your string variables have object methods, and even a hard-coded string has those same methods. The same is true of numbers, Arrays, and, of course, all the built-in objects you can access just by virtue of the fact that you're writing JavaScript code.
For example, Arrays have the `.join()` method which turns the Array into a string; and strings have the `.split()` method which turns the string into an Array.
And complementing those methods attached to the individual String, Number, and Array "objects" in your code, there are the Standard Built-in Objects. These objects' classes have methods that are sometimes uniquely useful, like `Array.isArray()` or `Object.keys()`.
Of course MDN has a [full list](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects) of all the built-ins, but it's probably more useful if I hand-pick a few: [Object](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object), [Number](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number), [Math](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math), [Date](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date), [String](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String), [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp), [Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array), [JSON](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON), and [Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise) are the ones I use the most.
### Arrays
If a variable is a box, then an Array is a box full of boxes. The internal boxes are numbered, starting with #0 for the first box. Those boxes can also be subdivided into boxes, as many layers deep as you need.
The following code sample contains a smattering of things you need to know about Arrays:
```js
let arr = []; // create an empty array
arr[0] = 'Arrays'; // set the 1st element
arr.push('rule'); // add an element at the end
console.log( arr[1] ); // output the 2nd element
if (arr.indexOf('rule') > -1) { /* do something if 'rule' appears in array */ }
let str = arr.join(' '); // make string with spaces between elements
```
#### Finding Your Way Around Arrays
Arrays never use dot notation -- they only use square brackets. Since you can have arrays within arrays, and objects within arrays, let's go into some example cases.
```js
const arrayOfObjects = [
// element 0: an object with info for the 1st Doctor
{ doctor: '1st', actor: 'Hartnell' },
// element 1: an object with info for the 9th Doctor
{ doctor: '9th', actor: 'Eccleston'}
];
console.log( arrayOfObjects[0].actor );
// outputs 'Hartnell'
```
Array element position numbers start at 0 and increment by 1 from there. So, `arrayOfObjects[0]` addresses the first element of `arrayOfObjects`. The first element of `arrayOfObjects` is an object, so we can address one of its fields by adding `.actor`.
Next, let's look at an array of arrays, aka a "multi-dimensional array".
```js
const arrayOfArrays = [
// element 0: an array of dark colors
['navy blue', 'forest green', 'dark grey', 'black'],
// element 1: an array of light colors
['sky blue', 'bright green', 'light grey', 'white']
];
console.log( arrayOfArrays[1][3] );
// outputs 'white'
```
Arrays can contain arrays that hold arrays *full* of arrays, on and on, with more and more dimensions of data complexity.
Later in this lesson, you'll see how to step through arrays using two types of loops designed specifically for that job. There are a lot more than two, though!
#### Array Methods
There are many other methods automatically attached to every Array object you create in your code. MDN has a [comprehensive guide](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array), of course. To get you started, here are a few key Array methods.
##### `.indexOf()` and Other Search Methods
Assume you have an Array called `arr` and you want to know if something is in that Array, or even want to know *where* it is in that Array. The code `arr.indexOf(thing)` returns the index of `thing` in Array `arr`. When you're using `.indexOf` simply to test whether the `thing` is in the Array, remember, Arrays start at index 0, meaning a 0 is a hit. So in a conditional like that, think like this:
- **-1** is false (the `thing` is **not** in the Array),
- **greater than -1** is true (the `thing` **is** in the Array).
```js
let arr = ['Arrays', 'rule'];
if ( arr.indexOf('rule') > -1 ) {
// index greater than -1 = hit
}
```
There's also `.find()` which lets you specify a testing function. `.find()` returns the first Array entry that makes your testing function `return true`.
```js
let arr = ['Arrays', 'rule'];
let match = arr.find( /* begin testing function */ (item) => {
// testing function code
return item.startsWith('Array');
} );
console.log( match ); // outputs "Arrays"
```
The Array method `.lastIndexOf()` is just like `.indexOf()` but it starts its search at the end of the Array and steps backwards until it finds a match. It returns -1 if no match was found, or if a match was found it returns the index of the found match.
##### `.push()` and Other Insertion Methods
The `.push()` method inserts one or more new elements at the end of the Array.
```js
const arr = ['Arrays', 'rule'];
arr.push('and');
arr.push('objects', 'do', 'too'); // add multiple
console.log( arr.length ); // outputs "6"
```
The `.unshift()` method adds one or more new elements at the beginning of the Array.
```js
const arr = ['Eccleston', 'Tennant', 'Smith', 'Capaldi', 'Whittaker'];
arr.unshift(
'Hartnell', 'Troughton', 'Pertwee', 'Baker',
'Davison', 'Baker', 'McCoy', 'McGann'
);
console.log( arr );
/* output:
[
'Hartnell', 'Troughton', 'Pertwee', 'Baker', 'Davison', 'Baker', 'McCoy',
'McGann', 'Eccleston', 'Tennant', 'Smith', 'Capaldi', 'Whittaker'
]
*/
```
Finally the `.splice()` method. This method is versatile: it deletes, it inserts, it'll chop off half the array or more... all depending on how you use it. For insertion at some arbitrary point in the Array, you need the index for the new entry (or entries). Anything currently at that position or later, will be bumped. Let's say you have the index in the variable `index`... you would do:
```js
const arr = ['apple', 'banana', 'grape'];
// let's suppose a variable index === 2
arr.splice(index, 0, 'canteloupe', 'dragonfruit');
console.log( arr );
// outputs:
// [ 'apple', 'banana', 'canteloupe', 'dragonfruit', 'grape' ]
```
That second argument to `.splice()`, which is set to 0 above, is the number of elements you want to delete at that position. So you can replace, insert, do a mass delete and an insert at the same time, etc. You can also do deletions *without insertions* using `.splice()`, because the list of new entries is an optional parameter. And, you can chop off every element after `index` using `.splice()`, by leaving off the second parameter.
##### `.join()` and Other Misc Methods
The `.join()` method turns an Array into a string, using a string as a linker between elements. Example:
```js
const arr = ['Arrays', 'rule', 'and', 'so', 'do', 'objects'];
console.log( arr.join(' ') );
// outputs "Arrays rule and so do objects"
```
The `Array.isArray()` static class method is how we deal with the fact that the following code outputs the string, "object":
```js
let x = [1, 2, 3, 4, 5];
console.log( typeof x );
// outputs: "object"
```
(`typeof` will be covered later this lesson; for now just understand it outputs a string, equal to the category of whatever's to the right of the word `typeof`.)
So we can't use `typeof` to find out if something is an Array. We have to use `Array.isArray()`. It's not `theVariable.isArray()`, it's `Array.isArray(theVariable)`, because this is a static class method rather than an object method.
The `.map()` method creates a new Array whose elements are generated by a callback function. You provide the callback function, which is run once per element in the Array. The callback function automatically receives inputs so long as you set the function signature to receive them. The usual signature is either `(item)` which gives you the element (each one, one iteration at a time), or `(item, index)` which gives you the element plus its index in the Array.
```js
const arr = [1, 2, 3, 4, 5];
const a = arr.map( (item) => {
return item*5;
});
console.log( a );
// output: [ 5, 10, 15, 20, 25 ]
```
The `.filter()` method copies a portion of an Array. You provide a function as the first (and usually only) argument to `.filter()`. This function gets run once for every element in the Array. The function receives input automatically, you just have to set your signature to handle those inputs. The two most commonly used signatures are `(item, index)` where `item` is the element and `index` is its index; or `(item)` -- same as before, minus the index.
```js
const arr = [1, 2, 3, 4, 5];
const a = arr.filter( (item, index) => {
if (index > 0 && index < 4) return true;
else return false;
});
console.log( a );
a[0] = ' Swiss';
console.log( arr );
```
Interestingly, the above script didn't perform the same exact way across JavaScript implementations. For one thing, the documentation for `.filter()` says the copy remains linked to the source Array, and therefore changing an element in one Array, should change the other; but the two Arrays were independent in both Node.js and Opera Browser (which is a fork of Chrome, under the hood). Also, my browser assigned `a[0]` the value of `"Swiss"` before outputting the content of `a`; that is, the `console.log()` method was somehow delayed, but only in the browser. I have no explanation for this quirk and my only advice is, test often.
#### Moving On
Arrays do a lot more than what I've demonstrated here (we'll cover one more Array method later this lesson). Always remember you can learn more reading the [MDN reference for Arrays](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array), just as soon as you get comfortable reading its more technical format.
### How to Find your Way around Objects
Just like with Arrays, it's all fantastic that an object can hold multiple values, but that does nothing for you if you don't know how to access the data inside.
#### Objects & Sub-Objects
Let's say you have an object like this:
```js
const fruitColor = {
banana: "yellow",
apple: "red",
pear: "green",
}
```
I think we all know by now that this will output `red`:
```js
console.log( fruitColor.apple );
```
You can continue chaining dots and field names so long as necessary to address nested objects:
```js
const fruitColor = {
banana: "yellow",
apple: { fuji: "red", grannySmith: "green" },
pear: "green",
}
```
To output `red` from the above data, we now:
```js
console.log( fruitColor.apple.fuji );
```
#### Finding Object Properties Using Variables
But what if the choice of fruit was unknown until runtime? What if you needed to use a variable to select into the `fruitColor` object, to identify which property you want? There is an alternate syntax for that:
```js
let whichFruit = 'banana';
console.log( fruitColor[whichFruit] );
```
Since `apple` was broken down to `fuji` and `grannySmith` in a sub-object, dynamic addressing gets a little more involved. If we want to output `red` from that data:
```js
let whichFruit = 'apple';
let whichType = 'fuji';
if (whichType)
console.log( fruitColor[whichFruit][whichType] );
else
console.log( fruitColor[whichFruit] );
```
You can mix and match dot notation vs square brackets as needed. For example, these are both the same thing:
```js
console.log( fruitColor.apple['fuji'] ); // red
console.log( fruitColor['apple'].fuji ); // red
```
But this doesn't work:
```js
console.log( fruitColor["apple.fuji"] ); // undefined
```
***By the way,*** you might notice I left off the curly braces on the `if`/`else` above. Any control flow feature that involves curly braces (so, almost all of them) can be done without the curly braces -- in which case, only one statement will happen (or not happen) as a result of that control flow feature. This is illustrated in the example code above: both the `if` and the `else` each have only one line indented under them. (That indentation is only to remind the programmer about the *scope* of each block (or line) of code -- indentation isn't a flow control feature in JavaScript.)
#### A Peek Ahead
Finally, looping through objects. We're getting ahead a bit, but we'll come back to it later this lesson. You use the `Object.keys()` method to get an Array of the keys, and then you loop through that Array, using the values you find along the way to select into the object. Like this:
```js
const obj = {
ball: "round",
seeSaw: "flat",
truck: "complex"
};
const keys = Object.keys( obj );
keys.forEach( (item) => {
console.log( obj[item] );
});
/* output:
round
flat
complex
*/
```
### Comparison Operators & Other Related Operators
So far we've only discussed one of the "comparison operators" -- that being `===` the Strict Equality operator. But there are many operators you can use to make a conditional:
<table>
<tr><th>Operator</th><th>Effect</th><th>Example</th></tr>
<tr><td style="padding: 4px 20px 0 0"><code>===</code></td><td style="padding: 4px 20px 0 0">returns true if values are equal and of equal type</td>
<td style="padding: 4px 0 0 0"><code>if (x === y)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code>==</code></td><td style="padding: 4px 20px 0 0">returns true if values are equal after type conversion</td>
<td style="padding: 4px 0 0 0"><code>if (x == y)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code><</code></td><td style="padding: 4px 20px 0 0">returns true if number to left is smaller than number to right;<br/>or if string to left comes alphabetically earlier than the string on the right</td>
<td style="padding: 4px 0 0 0"><code>if (x < y)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code>></code></td><td style="padding: 4px 20px 0 0">returns true if number to left is larger than number to right;<br/>or if string to left comes alphabetically after the string on the right</td>
<td style="padding: 4px 0 0 0"><code>if (x > y)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code><=</code></td><td style="padding: 4px 20px 0 0">returns true if value to left of operator is less than or equal to the value on the right</td>
<td style="padding: 4px 0 0 0"><code>if (x <= y)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code>>=</code></td><td style="padding: 4px 20px 0 0">returns true if value to the left is greater than or equal to the value on the right</td>
<td style="padding: 4px 0 0 0"><code>if (x >= y)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code>!==</code></td><td style="padding: 4px 20px 0 0">returns true if the two values are not equal or don't have equal type</td>
<td style="padding: 4px 0 0 0"><code>if (x !== y)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code>!=</code></td><td style="padding: 4px 20px 0 0">returns true if the two values are not equal after type conversion</td>
<td style="padding: 4px 0 0 0"><code>if (x != y)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code>in</code></td><td style="padding: 4px 20px 0 0">returns true if the string on the left of the operator is a key in the object to the right of the operator</td>
<td style="padding: 4px 0 0 0"><code>if ("message" in error)</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code>typeof</code></td><td style="padding: 4px 20px 0 0">returns a string describing the type of the variable after the operator</td>
<td style="padding: 4px 0 0 0"><code>if (typeof str === 'string')</code></td></tr>
<tr><td style="padding: 4px 20px 0 0"><code>instanceof</code></td><td style="padding: 4px 20px 0 0">returns true if the class reference on the right, is anywhere in the ancestry of the object on the left</td>
<td style="padding: 4px 0 0 0"><code>if (obj instanceof Object)</code></td></tr>
</table>
### `for` loops and the `Array.forEach` Method
I'm going to be honest with you, dear students. I don't know a single one of the newfangled `for` loops, just what I call the "traditional `for` loop". It looks like this:
```js
for (let i = 0; i < something.length; i++) {
// looped code
}
```
As you can see in the parentheses above, the `for` statement has three arguments separated by semicolons:
- The first one happens when the loop is about to begin
- The second one is a conditional that determines whether the loop happens, or happens again
- The third one happens after each run of the loop
So in the example above:
1. A variable called `i` is declared and defined as `0`.
2. Then `i` is compared to something called `something.length`. Arrays have a `.length` property that contains the number of entries in that Array. Remember, Arrays start numbering their entries at 0; this is why we're checking for `i < something.length` (`i` *less than* `something.length`) instead of `i <= something.length` (`i` less than *or equal to* `something.length`).
3. Then the `// looped code` area happens (which presumably contains actual code instead of just a one-line comment).
4. `i++` happens, which adds 1 to the stored value of `i`.
So how does this control flow structure help you walk through an Array and address every element one by one? Because it gives you a variable (named `i`) that's going to loop every whole number from 0 to that Array's `.length - 1` That's perfect:
```js
// the next line creates an array
let something = ["truck", "train", "aeroplane"];
// as long as i is less than 3, keep looping
for (let i = 0; i < something.length; i++) {
// we're in the loop now
// something[i] is now an element of something
// output that element
console.log( something[i] );
}
```
But there's a shorter way that's great in most situations. As long as you don't need to `return` anything, you can:
```js
let something = ["truck", "train", "aeroplane"];
something.forEach( (item, i) => {
// looped code goes here
// item is the element of something
// i is the "index" of something
console.log( item );
});
```
As shown above, `Array.forEach` takes a callback as an argument. The callback will be called once for each element of `something`. Each time the callback is called, `item` and `i` will be pre-filled with the next element of `something` and the index of that element. (The "index" is the number you'd put between square brackets to get that element.)
### The Increment (`++`) and Decrement (`--`) Operators
You'll have noticed that the traditional `for` loop has `i++` as the third statement in its funky signature. This is the "Increment Operator". It adds 1 to the variable it's attached to, assuming it's a number. You can subtract 1 from a variable just as easily, which for example is how you'd do a `for` loop that counts backwards: `i--`.
### `while` Loops
Another situation you'll occasionally find yourself in that calls for the Increment Operator in a loop: you want a loop that runs until a condition is met. But you also want a limit to how many times the loop will run -- rather than strictly waiting for that condition to change. This is a job for the `while` loop.
```js
let isDone = false;
let i = 0;
while ( !isDone && i < 10000 ) {
isDone = checkStatus();
i++;
}
```
We know the `while` loop will run at least one iteration because `i` starts at 0 and `isDone` starts as `false`. Why does that mean at least one iteration? Because after `while` is a conditional that says: evaluate as `true` **if** `isDone` is `false` or evaluates to `false`, **and** `i` is less than 10,000. If the conditional evaluates as `true`, then the loop runs. After that loop, it checks the conditional again, and, if the conditional still evaluates to `true`, the script runs the loop again.
We got this result out of the `while` conditional thanks to some new things: the "Logical Not" operator, and the "Logical And" operator (plus we used the "Less Than" comparison operator, covered earlier this lesson). Logical Not inverts whether something will evaluate as true. Logical And allows you to chain together conditional logic; the condition on the left and the condition on the right must both evaluate as true in order for the conditional to evaluate true as a whole.
Inside the loop we get the return value of some function called `checkStatus()` which we can assume exists. Hopefully, whoever programmed the code above, checked the return value data type first, and confirmed it eventually outputs something that evaluates as true! We store the return value of `checkStatus` in the variable `isDone` which is the same variable checked in the `while` conditional. Then we increment `i`. Here the loop ends, with control flow now hitting the `while` conditional to see if it's still true. If it's false, control flow skips to the next line after the `while` loop's closing bracket. But if the `while` loop's conditional evaluates as `true` forever, the script will keep running that loop forever, or until someone or something shuts it down.
### Lesson Conclusion
Manipulating and navigating your data is the essence of programming. Every category of software must manipulate data, and to do that, often, a coder must navigate data structures. There are often situations such that, not only do you have a complex data structure, but you have an array *full* of those structures. Therefore, it's important to understand loops, and know which loop is best for which job. Remember, the traditional `for` loop might be slightly more cumbersome to write out, but it lets you `return` effectively. Today's lesson also unlocked the conditional part of `if` and `while`, by exposing the conditional operators. And not last but least, I explained what the `++` and `--` operators do.
How to Code: The Fundamentals, pt 0
By AstroMacGuffin dated Sat Aug 06 2022 15:16:25 GMT+0000 (Coordinated Universal Time) last updated Sat Aug 20 2022 17:52:38 GMT+0000 (Coordinated Universal Time) With parts 1 and 2 of this series already out, I now realize I skipped a few topics crucial to many students. Things like:
- how to run JavaScript code,
- how to find bugs in JavaScript code,
- what is Node.js,
- what's the difference between Node.js and JavaScript,
- what editor should I use to code in JavaScript,
- what is HTML,
- what is DOM,
- what is an NPM package,
- etc
This post is for beginners, and those whose experience mainly consists of jumping in too deep with not enough fundamental knowledge. Let's get you sorted on the bare beginnings of how to code in JavaScript.
Featured Article
How to Code: The Fundamentals, pt 2
By AstroMacGuffin dated Thu Aug 04 2022 19:22:18 GMT+0000 (Coordinated Universal Time) last updated Wed Aug 17 2022 11:22:33 GMT+0000 (Coordinated Universal Time) If you finished part 1 of this series and thought to yourself, "that wasn't enough for one lesson, I need more immediately!", then you're an odd one, because that was a lot of information. But here we go again! After a quick roundup of smaller topics we're going into objects, classes, and async/await (aka asynchronous code)!
### This Lesson
First a round-up of important topics that don't fit elsewhere: `null` and `undefined` as both data types and values; and Garbage Collection aka GC -- JavaScript's technique for memory management. Then I explain everything a beginner needs to know about asynchronous code aka `async`/`await`. Then we jump headfirst into objects and classes.
Until recently, programming meant writing lines of code in full confidence that, when you run that code, line 1 would finish before line 2 would begin. And, today, it's still mostly the same. But then a rookie slams headfirst into an asynchronous function. Today we're going to treat `async`/`await` as a fundamental.
It's also time to cover objects and classes. Object Oriented Programming is probably the single most powerful tool in rapid programming ever. You'll learn how to treat code like layers in an image, stacking them on top of each other to create an end result that can save a professional programmer hours per task.
### `null` and `undefined` Data Types
In programming there are multiple ways to say "nothing". The `null` and `undefined` data types correspond to the built-in `null` and `undefined` values. The value `null` is nothing special... it's just "nothing". Meanwhile `undefined` can mean the variable has been created, but hasn't yet been assigned a value. (Or, it can also mean someone deliberately set a variable to equal `undefined`.)
Both `null` and `undefined` evaluate to `false`. As a matter of style, I tend to set my variables to `undefined` when I need to signify they're in a "reset" state; and I tend to return `null` (or just `false` itself) from functions if some criteria wasn't met.
You can use the `typeof` operator to see `if (typeof someVariable === 'undefined')` but it's quicker to check the value: `if (someVariable === undefined)` loses both the `typeof` operator, and the quotes on `undefined`.
### Garbage Collection
Memory management is hard, and JavaScript is neither the best nor worst at it. Its primary means of keeping a trim footprint is called "garbage collection" or GC. The GC system relies on you, the programmer, to keep a clean house in your code.
Data disappears from your program, freeing up memory, when there are no more variables pointing to it. Remember from part 1 of this series: any data type that reports itself as an object (or an Array) gets variables that all point to a single instance of the data. It takes special techniques to ***copy*** an object or Array. If your code says `const whatever = someObject;` you now have one copy of someObject, referenced from two different variable names.
This matters because, someday, you'll be building indexes of data. You'll need to be able to clear individual entries from those indexes, to keep your programs memory-efficient.
### Async/await - Asynchronous Functions
Our most basic control flow pattern -- top line to bottom line, one line at a time -- is no longer guaranteed, thanks to modern programming features. Programs can now run functions asynchronously, which means functions can activate and do whatever, ***but your outer scope code doesn't wait for the function to finish before continuing with the next line of code.*** As a beginner, you need to understand this fundamental side of the language: the asynchronous side. Because, someday soon, you will do something like:
```js
let someVariable = someFunction( options );
```
...and everything will seem fine, but you won't receive any data in `someVariable`... or if you do have any data in someVariable, it's some object called a Promise. In this situation you should check the declaration of `someFunction` to see if it is `async`. An `async function`'s declaration looks like one of these:
```js
async function someFunction( options ) {
// code
}
// in the following forms, let or var may be used in place of const
const someFunction = async ( options ) => {
// code
}
// or
const someFunction = async function( options ) {
// code
}
// object classes have methods, which don't use the "function" keyword, but they're totally functions
async someMethod( options ) {
// code
}
// another way of declaring an object method
someMethod: async function ( options ) {
// code
}
// yet another way to declare an object method
someObject.someMethod = async ( options ) => {
// code
}}
// ...or...
someObject.someMethod = async function( options ) {
// code
}}
// etc... yes, there are still other ways...
```
#### Solution 1: `await` the `async`
If it's `async`, you should adjust the code where you call that function. Instead of
```js
let someVariable = someFunction( options );
```
...you add `await` before the function call, e.g....
```js
let someVariable = await someFunction( options );
```
**This won't solve every situation.** But in many cases, a quick `console.log()` will confirm you now have the data you expected, instead of a Promise. If it doesn't solve your situation, then you probably need to know about "Thenables" and Callbacks.
#### Solution 2: Callbacks: Functions as Function Input
A callback is a function you pass into another function as input. I don't mean like this:
```js
y.setHeight( x.getHeight() );
```
That's passing the function's *output* as the input of another function. I'm talking about passing the function *itself* as input. Remember, everything is a variable, even a function:
```js
y.setHeight( x.getHeight );
```
What's the difference? No parentheses on the function being inputted as a callback. Why do this? Storing the function gives you the ability to re-run the callback, giving you a fresh return value based on up-to-the-picosecond data.
There's also the anonymous callback. An anonymous function or anonymous callback is a nameless function made up on-the-fly. So yes, you can actually *write a function in the middle of calling another function.* That is a very common practice actually, and would look like this:
```js
aFunctionThatExpectsACallback( () => {
// code
} // end of callback
); // end of call to aFunctionThatExpectsACallback
```
Many asynchronous functions don't `await` properly and their programmers worked around this by accepting a callback. The callback is run when the job is done.
#### Solution 3: "Thenables" & Promises
Thenables are objects that have a `.then()` method which takes a callback as an argument. The callback happens when the Thenable finishes its other tasks. The standard JavaScript "Promise" object is a Thenable. So here is another way to force your code to wait until the Promise has turned into a value:
```js
// get the promise
let foo = await anAsyncFunction();
// now we must wait for the promise to resolve:
// method 1: declared function as callback
function someHandlerFunctionForWhenItsDone(data) {
// do stuff with data
}
foo.then( someHandlerFunctionForWhenItsDone );
// method 2: anonymous function as callback
foo.then( (data) => {
// do something with the "data" variable
});
```
As shown in "method 2" just above, the callback will usually have at least one parameter, which is your output data, automatically populated into the callback's input when the task is done. The spec for any given callback will also often have an error object as an input parameter; again, this is auto-filled when your callback starts to run.
#### In Conclusion (For Now) regarding Asynchronous Code
Asynchronous features are a good thing. They make interfaces faster. But as a beginner, they can seem like voodoo and a curse. Just remember there's always a way to get the result you want; the three techniques above will get you through basically all situations.
### Objects
If you're one of those students who started with hands-on tutorials before seeking to learn the basics, then you've probably used objects before. Anytime you see the dot notation such as `Array.isArray()` or `someString.split()` or `Object.keys()` that's either an object, or a class.
Sharp-eyed students may have noticed I included strings as objects! But in part 1, strings were one of the data types ***not*** counted as an object! On a certain level, everything is an object in JavaScript; the two categories of data types from lesson 1, the "primitives" vs "objects", still stands as the line between values being copied vs objects being referenced. Try not to think about it too hard.
Choose any real-world object in your line of sight. It has properties: it has at least one color, at least one shape, is it shiny or dull, is it powered, is it rechargeable, and so on. In programming, that's your most basic object. An object is a variable type. When used at this simplistic level, its equivalents in some other languages include the dictionary, or the hash, or the associative array. The most basic JavaScript object is just a variable that holds sub-variables. You can define one very easily on the fly, almost anywhere in your code:
```js
const ball = {
color: 'red',
bounciness: 1.0,
size: 'large',
};
```
Notice this is basically identical to JSON, the **J**ava**S**cript **O**bject **N**otation language.
This kind of object is very useful, but not very powerful. What if it could handle its own bounce command? Of course, it can:
```js
const ball = {
color: 'red',
bounciness: 1.0,
size: 'large',
isBouncing: false,
bounce: (milliseconds) => {
// bounce for a number of milliseconds
console.log('The ball starts bouncing.');
ball.isBouncing = true;
setTimeout( (obj) => {
console.log('The ball stops bouncing.');
obj.isBouncing = false;
}, milliseconds, ball);
},
};
// Set it off
ball.bounce(1000);
// Monitor status
console.log(ball.isBouncing);
setTimeout( () => {
console.log(ball.isBouncing);
}, 1500);
```
But what if you need your ball code to be reusable? You create a **class**. Then your code can say something like this:
```js
const zipper = new Ball( { color: 'red', bounciness: 1.0, size: 'large' } );
zipper.bounce(1000);
```
### Object Classes
A class is like a second data type, for objects. Writing class-based code unlocks new features in object-oriented programming, such as using `this` to mean "this instance of the class as an object".
Making object classes is easy:
```js
class Ball {
constructor(arg) {
this.color = arg.color;
this.bounciness = arg.bounciness;
this.size = arg.size;
}
bounce(milliseconds) {
// bounce for a number of milliseconds
console.log('The ball starts bouncing.');
this.isBouncing = true; // "this" instead of "ball"
setTimeout( (obj) => {
console.log('The ball stops bouncing.');
obj.isBouncing = false;
}, milliseconds, this); // "this" instead of "ball"
}
}
const zipper = new Ball({color: 'red', bounciness: 1.0, size: 'large'});
zipper.bounce(1000);
```
Important but dumb thing about terminology: A function that resides inside an object or object class, is called a "method". Methods are functions. Go ahead, do a `console.log(typeof zipper.bounce)` and see if it doesn't confirm - methods are functions. Almost everything that's true about methods is true about functions, and virtually everything I'll ever say about functions also applies to methods. They're the same thing (so why couldn't they have the same name?) ...
The *method* `constructor` runs whenever there's a `new Ball()`. The `constructor` method receives the arguments given to `new Ball()`. Here we have it configured for a single argument:
```js
class Ball {
constructor(arg) {
```
And when `new Ball()` is called, we create an object on the fly to serve as the single argument:
```js
const zipper = new Ball( {color: 'red', bounciness: 1.0, size: 'large'} );
```
#### Extending Classes
Remember the analogy about putting code on top of code, like layers in an image? This is where you do that. Consider if we rework the `Ball` class like so:
```js
class Toy {
constructor(arg) {
this.name = arg.name;
this.color = arg.color;
this.size = arg.size;
this.brand = arg.brand;
this.cost = arg.cost;
this.owner = arg.owner;
}
}
class Ball extends Toy {
constructor(arg) {
super(arg); // call the parent's constructor
this.bounciness = arg.bounciness;
}
bounce(milliseconds) {
// code
}
}
```
The `extends` keyword creates a parent/child relationship between two classes. The line `super(arg)` in the `constructor` for `Ball` refers to the `constructor` of `Toy`. In this relationship, `Toy` is known as the "parent class". The "child class" is `Ball`. The child class absorbs all the properties of the parent, including methods.
But that's not all. The new child class is also linked to the parent of the parent class, and the whole ancestry, leading back to a class that doesn't extend anything. Every property and method from that family line, gets absorbed into the child class.
Anytime the same method name appears in two or more classes in the family line, the child class overrides. The child class method can (and often should) call the parent class method, similarly to how the parent's constructor was called from the child's constructor:
```js
class Toy {
constructor() {
}
setColor(color) {
this.color = color;
console.log('The color ' + color + ' was saved.');
}
}
class Ball extends Toy {
constructor() {
super(); // this line is mandatory in child constructors
}
setColor(color) {
super.setColor(color);
console.log('The ball changed colors!');
}
}
```
`super.setColor()` calls the `setColor` method on the parent class. It's almost always a good idea to put the input from the child method, into the input for the parent method, as I've done in the example:
```js
setColor(color) {
super.setColor(color);
```
Now that we've swept through the most powerful features of objects and classes, let's wind this lesson down with two more class features that are often overlooked by rookie programmers. Both of them deal with the keyword `static` and how it modifies properties and methods.
#### Classes as Data Tables
When you normally give properties to a class, you do so by defining the properties in the class constructor. (Yes, you might declare or even define normal class properties above the constructor, but sometimes you will have more complex initialization for a property than what is allowed in that format, and why divide up your initialization between two parts of the code?) This makes it so the property is associated with an *instance* of that class, aka an object of that class. You can't use those properties until you create a new instance of the object:
```js
const zipper = new Ball( {color: 'red', bounciness: 1.0, size: 'large'} );
console.log(zipper.color);
```
But it's also possible to put data into a class in such a way that you don't need an object of that class to access those properties. These are called "static properties". Consider:
```js
class Toy {
static purpose = 'fun';
}
console.log(Toy.purpose);
```
This is a great place to put configuration flags, error messages, and so forth. Don't forget that these properties can be objects, so you might wind up with:
```js
class Toy {
static ERR = {
UBROKE: 'You broke it!',
NOTFUN: "You don't find it fun anymore.",
PENYWS: 'The toy fell into a sewer grate!'
};
}
// throw an error: we don't want to play with Pennywise
throw( Toy.ERR.PENYWS );
```
#### Classes as Function Libraries
The same trick works on methods. Functions that seem logically grouped within a class's overall concept, are excellent candidates for static methods in that class. Perhaps we had already written a `throw()` function before creating the `Ball` class.
```js
function throw(obj, thrower, target) {
console.log(thrower + " throws the " + obj + " to " + target + ".");
obj.owner = target;
}
```
It might make more sense for this to become an object method, depending on your project. But perhaps we want the code to be capable of throwing non-ball objects *in the manner of a ball*. So we move `throw` to the `Ball` class with just one tiny change:
```js
static throw(obj, thrower, target) {
console.log(thrower + " throws the " + obj.name + " to " + target + ".");
obj.owner = target;
}
```
#### When to Use `static` and When Not To
There's two simple guidelines about whether or not your method or property ought to be a static:
- Needs an Object: if the method needs to read from, or write to, live object properties, then the method *probably* shouldn't be static. (However, don't forget the example above for `Ball.throw`, where the object was passed in as an argument, making the code more powerful because it can operate on anything.) Same for properties: if a property describes a specific object, it shouldn't be static. There is no exception for properties. Static properties should describe **all** objects of the class. You wouldn't make `size` a static property of a Ball class, but you might make `shape` a static property equal to "round" (if you wanted to troll American football fans).
- Object More Convenient: Soon it will be common for you to encounter *objects full of objects* as you code. Normally each class goes in its own file called a module; to use a static class feature or make a new object of a class, you have to go back to the top of your file and `require` that class module. It's a minor hassle and totally avoidable if an object of that class is already conveniently available in your code's current context. In that case, common sense favors making the method or property *not* static.
### Lesson 2 Conclusion
I know we're going through these topics at breakneck speed. Pace yourself through each lesson as you see fit. I think coding is exciting, and I want these lessons to be here for people who want to binge this education.
Featured Article
How to Code: The Fundamentals, pt 1
By AstroMacGuffin dated Thu Aug 04 2022 11:24:00 GMT+0000 (Coordinated Universal Time) last updated Sat Aug 13 2022 12:12:45 GMT+0000 (Coordinated Universal Time) As moderator and volunteer on the JavaScript Mastery discord, I've noticed a trend. The people who are in the most distress when they come for help, are the people who jumped head-first into a tutorial that was far too advanced for their skill level. They don't even know the fundamentals of JavaScript and they're trying to build a TikTok clone or a medical-grade messaging app.
Some of them don't even want to learn, and are eager to say so.
For the rest of you: welcome back to basics. There's a lot of fundamental knowledge you may have missed. If you're not really sure what variables are; if you have no idea how to create a function; and if you've never heard of the terms "control flow" or "scope" before, everything you do will make more sense after you grok this information.
If someone (such as myself) sends you this link, congratulations! That person thinks you'll be a better programmer with just a little bit of high-density reading!
People who don't write code might think it's magic, but it's just language, and logic. Anyone can pick it up, because our species excels at both language and logic. So let's do this!
If you feel like I skipped some crucial information, maybe try [part 0 of "How to Code: Fundamentals"](https://astromacguffin.com/ref/id/62ee85c9ae620ee833fd88b7).
### This Lesson
First you'll learn about control flow and scope. Then you'll learn about variables, a little bit about data types, and then functions.
Control flow is a programming concept that means you are in control of which instructions get done in what order. In this lesson we'll mostly be focused on the simplest control flow scheme: line 1 happens, then line 2 happens, then line 3, etc. until the program is complete. I'll use the `if` statement to demonstrate a common control flow feature of programming.
Variables are nametags for pieces of data. There are many types of data. Variables are not "typed" in JavaScript, meaning a variable can point to data of one type one moment, and another type of data the next.
Functions are nametags for code, but they're also more. Functions take input and send output, and can be any level of complexity, making them entire mini-programs within your program.
### Control Flow
"Control flow" refers to things you can do in programming that control which instructions get sent to the CPU in what order. Take your simplest code sample, a traditional Hello World in browser-side JavaScript:
```js
const text = 'Hello World';
alert(text);
```
It has two lines of code:
1. The first line of code creates a `const` -- a "constant", which is a type of variable. The `const` named `text` will hold the text, "Hello World".
2. The second line of code calls a function that's built into JavaScript, `alert`. It passes input to that function by placing the variable name `text` between the parentheses.
`alert` is a well-known function that pops up a simple message box on the web browser. What should the message say? That's the input you provide when you call the `alert` function. More on functions later. The upshot for now is that, when you run this code, a pop-up, saying "Hello World", should appear.
But how does the `alert` function know anything about a variable named `text`? That's thanks to control flow, and scope, a second feature under your control, which will be explained soon. Suffice to say for now: control flow goes from top to bottom, line by line. Since the definition of `text` came before the call to `alert`, `text` existed when `alert` was called.
To explain scope better I'll introduce the `if` statement. This code contains a common mistake:
```js
let color = 'blue';
if (color === 'blue') {
const text = 'Hello World';
}
alert(text);
```
What's happening here? First, let me explain *operators*. A single `=` equals sign is known as the "assignment operator". It assigns a value on the right hand side, to a variable on the left. So on the first line of code above, a variable called `color` is being *assigned* the value "blue".
But on the next line of code, we have three equals signs `===` back to back. This is one of the many *comparison operators*. The `===` operator means "is exactly equal to".
And of course, on that second line of code above, we've met our first `if` statement. If the conditions in the parentheses evaluate to `true`, then the code in the curly braces will be executed. This is a control flow feature. If the code above was run with no modifications, the following line of code would always be executed because `color` will always be exactly equal to "blue" (since we just defined it as such in the line above):
```js
const text = 'Hello World';
```
But since that's all the code in the curly braces after the `if` conditional... what happens next? After any control flow structure in your code has met its end, control flow always resumes its previous state, meaning in this case, we go back to doing things line-by-line, top to bottom...
### Scope
...Which would be great if not for the bug. It's a very common bug: a variable is being used "out of scope".
Let's look at the whole thing again:
```js
let color = 'blue';
if (color === 'blue') {
const text = 'Hello World';
}
alert(text);
```
If you ran it in a browser, no pop-up box would appear, and you'd be told `Uncaught ReferenceError: text is not defined` in the Developer Tools console. (To open the console, right click almost anywhere in the page, and choose **Inspect** -- e.g. **Inspect element**.)
So why isn't `text` defined? It's because it fell out of scope. Basically, each set of curly braces creates a context one level "inner" compared to the context of where you typed those new curly braces. Using our example, the `const text = 'Hello World';` line is in a context one level "inner" vs everything outside those curly braces. **Each of these contexts has its own scope.**
Variables from "inner" levels of scope can't be used in "outer" levels of scope. Variables from "outer" levels of scope **can** be used in the "inner" scope code. Just remember, in any given situation, the so-called "outer scope" is often an "inner scope" compared to the overall global scope, that is, the outermost part of the control flow. ***The variable will fall out of scope when the block that created it, ends.*** So long as that doesn't happen before control flow reaches the part of the code where you want to use the variable -- the variable will be available.
In the example code, the `if` block ended with the closing curly brace `}` and so the `const text` no longer existed when `alert` was called. That's scope.
### Variables & Data Types
Variables (aka vars) are nametags for data. This alone is a powerful feature. Combining variables with other variables, and combining variables with functions, represents almost 100% of programming, so this section is critical.
Someone recently asked in the JavaScript Mastery discord why variables were important, so here's a bullet list of reasons off-the-cuff:
- Configuration (the easiest config system for any piece of software is a list of config vars at the top)
- Software Speed (when the same value is referenced multiple times, a variable is faster)
- Function Arguments (without which functions would be far less powerful)
- Readability (even if you're the sole maintainer of a program, you'll thank yourself later if your code is readable)
- Debugging Steps (easier to find the mistake in your code if that mistake leaves a named imprint you can output)
First thing you need to know is the three types of variable, identified by the keyword used to declare them: `const`, `let`, and `var`.
- `const` - The variable must be defined. Can't be changed. Normal scope rules apply.
- `let ` - The variable need not be defined. Can be changed. Normal scope rules apply.
- `var ` - The variable need not be defined. Can be changed. Special scope rules apply.
#### Declaring & Defining Variables
In JavaScript, as with most languages, you have to *declare* a variable before you can use it. You declare a variable by issuing one of the above three keywords, plus the variable's name.
```js
let foo;
var bar;
```
You can go one step further in the same line, *defining* the variable by using the assignment operator and some value on the right of that.
```js
let foo = 2;
var bar = 'The quick brown fox jumped over the lazy dog.';
```
Variables made with `const` must always be declared and defined at the same time.
```js
const foobar = 'Two quick brown foxes jumped over three lazy dogs.';
```
#### Why `var` is Useless
So what are these "special scope rules" attached to `var`? I'll borrow the example from [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/var):
```js
var x = 1;
if (x === 1) {
var x = 2;
console.log(x);
// expected output: 2
}
console.log(x);
// expected output: 2
```
The special scope rule is that `var` does nothing special. You still have to declare the variable in the outside scope, and even the second `var` in the `if` block (`var x = 2;`) offers no special features. Compare the above, to the below:
```js
let x = 1;
if (x === 1) {
x = 2;
console.log(x);
// expected output: 2
}
console.log(x);
// expected output: 2
```
They're practically identical (the second example is shorter by one keyword) and they get the same result. For this reason, JavaScript coders tend to drift toward *never* using `var`. Just remember: `const` for constants that can't be changed, `let` for everything else.
In all cases, you need to remember to declare your variables in proper scope, so that the variable will be available when it's needed, and released from memory when it's not needed anymore.
### Data Types
Variables in JavaScript both do, and don't, have a "type". Every piece of data has a "data type", and yet, variables are not "strongly typed" in JavaScript -- meaning any variable can point to any type of data -- no variable is locked into the type of data it was first assigned. Still, data types are important; eventually, you will find yourself often querying the data type of a variable as part of an `if` or `switch` statement, to make more powerful functions. Long story short, we need a quick word on data types.
For a beginner, here's what you need to know about types. There are two categories.
Category 1: "primitives"
- Data type `boolean` stores either `true` or `false`
- Data type `string` stores text of whatever length
- Data type `number` will be reported for every number you'll ever use
- `null` and `undefined` data types, explained in the next lesson
Category 2: "objects"
- variables created using the `let obj = new className()` syntax
- variables created using the `let obj = { field: 'value', property: 1 }` syntax
- arrays, regular expressions, etc
Why care? Because category 2, the objects, are always handled "by reference". This means if you assign it to another variable, both variables point to the same object, as shown here:
```js
let foo = { field: 'value', property: 1 };
// foo.property is currently 1
const bar = foo;
bar.property = 2;
console.log(foo.property); // output is 2
```
Only the object types do this.
### Functions
Functions give nametags to code, but also much more. Functions can take input, and can send output, like little mini-programs. Most of the best functions are small, single-task functions.
Let's take a look at a simple function that has both input and output:
```js
function add(a, b) {
return a + b;
}
```
The function is called `add` and it adds two numbers, that are given to it when it's called (the inputs). It adds the two numbers, returning (outputting) the result.
Let's break this down. Obviously `function add` creates a function called *add*. The `(a, b)` is where things get interesting. This is called the function's "signature". This is how you define the inputs for the function. In this example, we're defining two variables, `a` and `b`, that will be available inside the function. The values of `a` and `b` depend on whoever writes code "calling" this function. (You "call" a function by typing its name with a set of parentheses after it.) For example, if someone writes:
```js
let c = add(6, 18);
```
...then, inside the function, `a` will equal 6 and `b` will equal 18. Notice that the person calling the function used the same number of inputs as the signature specified.
And the last thing to understand about the function definition for `add`, is the line starting with `return`. The `return` statement is a function's output. If the function is used thusly: `let c = add(6, 18);` ...you might notice the `let c =` part and wonder, how does it know what `c` should equal? It knows because of the `return` statement inside the function. This example, our `add` function, doesn't even create a variable to return -- it just returns the direct result of a math operation. Outputting values of whatever complexity in this way is fine.
Let's look at a slightly more complex example. Let's say you have a simple project for logging and storing log entries.
```js
const ERR_LOGSTOR = 'Invalid storage medium'; // an error message
const logStorage = []; // an array where we store our logs
// function signature with a default value
function storeLog(logMessage='Message not specified') {
if (Array.isArray(logStorage)) {
logStorage.push(logMessage);
return 0;
}
else {
return ERR_LOGSTOR;
}
function logMessage(logMessage=undefined) {
if (logMessage === undefined) {
console.log('An undefined message was not logged');
}
else {
console.log(logMessage);
const status = storeLog(logMessage);
if (status === ERR_LOGSTOR) {
console.warn('Log storage error: ' + status);
}
}
}
logMessage('System started');
```
Let's break that down:
```js
const ERR_LOGSTOR = 'Invalid storage medium'; // an error message
```
Whenever you see two slashes `//` it means that the rest of that line is a "comment". This is how programmers put documentation into the code itself. Code can also be "commented out" by putting two slashes at the beginning of the line. Code that's been "commented out" is completely ignored when you run the program.
```js
const logStorage = []; // an array where we store our logs
```
Two square brackets `[]` means "create a new, empty array". So when we begin this program, `logStorage` will be assigned an empty array.
```js
function storeLog(logMessage='Message not specified') {
```
This is the function declaration and the beginning of its definition. The signature is one value this time, but the value has a default. Since the value has a default, you can skip that part of the signature when calling the function: it could be `storeLog('I am a log message')` or it could be `storeLog()` with nothing between the parentheses. If nothing is between the parentheses, the function will proceed as if you had typed `storeLog('Message not specified')`.
```js
if (Array.isArray(logStorage)) {
logStorage.push(logMessage);
return 0;
}
else {
return ERR_LOGSTOR;
}
```
In the above snippet, first we do a sanity check -- meaning we check the code's situation to avoid producing an error. In this case, that means making sure the `logStorage` variable is an Array. If it is, we add the `logMessage` to the end of the `logStorage` array and output a 0 as our function's return value. If `logStorage` is not an Array, we do nothing except return the error message defined at the top of the script.
A quick additional note about `return`: using `return` also exits control flow from the function, in addition to sending output. Notice you can have multiple `return` statements in a function. Control flow will only arrive at one of them, but you can manage this with features like the `if` statement.
```js
}
```
This ends the function. Every code block begins with `{` and ends with `}`. This applies to control flow structures such as the `if` statement, as well as functions, classes, etc.
```js
function logMessage(logMessage=undefined) {
```
Once again we have a function declaration with a single parameter as its signature, and once again the parameter has a default value.
```js
if (logMessage === undefined) {
console.log('An undefined message was not logged');
}
```
Here we're doing some error trapping: we don't want to log nothing, but we do want to log the fact that nothing was logged. (Ah, business logic.)
```js
else {
console.log(logMessage);
```
Whenever you use `if`, you can use `else` right after the `if` block ends. The `if` block executes if the condition in parentheses is true; the `else` block runs otherwise.
```js
const status = storeLog(logMessage);
```
Here we're using the previous function, `storeLog`. Yes, you can use functions within functions. If doing task A means also doing task B, you'll probably find yourself calling one function from within another.
```js
if (status === ERR_LOGSTOR) {
console.warn('Log storage error: ' + status);
}
```
When we ran the `storeLog` function we also grabbed its return value. Now we're checking it against our known error messages, and if there was a match, we're warning the console operator there was a problem.
```js
}
}
```
Here we're closing the `else` case and closing the function.
```js
logMessage('System started');
```
Finally we set off the whole chain of events by calling `logMessage` to log a message that the system has started.
Move on when you're ready to [part 2 of "How to Code: The Fundamentals"](https://astromacguffin.com/ref/id/62ec1c6a209885383e63d857).
How to Sanitize Inputs for Web App Security in Node.js
By AstroMacGuffin dated Sun Jul 31 2022 10:09:57 GMT+0000 (Coordinated Universal Time) last updated Mon Sep 05 2022 10:20:46 GMT+0000 (Coordinated Universal Time) One of the friendly members of the JavaScript Mastery discord server did me a favor by performing some security auditing on this website. I admit, I was in a rush to launch, and I wasn't in any hurry to spend time on security steps. When I tried the `mongo-sanitize` NPM package it did nothing, so there went my lazy option. But I already had code for stripping symbols from a string, thanks to the search index / relevance-weighted search project. It just needed a little adjustment.
Once you have something that can sanitize inputs, you need to use it. And, because every input is different, there's no getting around this part - you have to analyze your input-handling code line-by-line for ways you can be hacked. That means inputs that:
- get used for database inputs and queries
- get used as filenames
- get used for logical control structures
Here's a brief primer from someone who can explain it like a newbie, because when it comes to security, I only know so much. In other words this is a starting point, not the end-all-be-all, when it comes to web app security.
Featured Article
How to Code a Simple Media Manager & File Uploader for a Node/Express Site in 2022
By AstroMacGuffin dated Wed Jul 27 2022 22:06:06 GMT+0000 (Coordinated Universal Time) last updated Sat Sep 03 2022 22:39:09 GMT+0000 (Coordinated Universal Time) The first thing you need to know about this media manager is that it's not a good idea to expose it to normal users. They'll be able to create folders, and the filename screening isn't good enough to trust the inputs from random people.
But if you need a way to get Markdown-formatted image tags easily from the images on your website, or just a working version of Express.js file uploads in 2022, as I did, this lesson will get you there. And if you need a springboard you can bring your own improvements to, as I've done since this article was first pushed... I enjoy working with it. I hope you will too.
### Ingredients
- a simple `MediaManager` object on the server side, to handle utilities such as setting/getting path names, creating folders, and getting lists of folders and files in a given subfolder
- a POST route in your Express.js app for handling all MediaManager requests (which will all be AJAX, via jQuery)
- middleware that acts on your route - in the form of an NPM package, called `multer`
- a pug template for the media manager interface
- some extra pug code in whatever interfaces where you want to enable the media manager
- jQuery and a plugin for jQuery, called jquery-uploadfile, which takes a ton of sweat out of the process and gives you a great UI experience including progress bars and options for multiple file uploads and drag 'n drop
- about 145 lines of CSS code
- a client-side JavaScript file to set up the media manager's interface buttons and so forth
### What This Media Manager Does
It's basically a file browser. You can navigate folders, and when you find the image you want, you click it; it then copies Markdown code to your clipboard which you can paste into an input or (more likely) a textarea.
It handles file uploads. The files will be uploaded to the folder you're viewing. It also creates new folders.
### What This Media Manager Doesn't Do
It can't delete files. I decided that, since this is a tool only for admins, and I'm going to be the only admin of this site for the foreseeable future, I can delete files another way.
### On To The Code! First Up: The Server Side Object
This is `media-manager.js`, a Node object:
```js
const path = require('path');
const fs = require('fs');
class MediaManager {
constructor(pathName=global.config.media_path) {
if (pathName.indexOf('..') === -1 && pathName.search(/^\//) === -1)
this.basePath = path.join(__dirname, pathName);
else
this.basePath = path.join(__dirname, global.config.media_path);
this.subPath = '';
}
setSubPath(pathName) {
if (pathName.indexOf('..') > -1 || pathName.search(/^\//) > -1)
return false;
this.subPath = pathName;
return true;
}
getPath() {
return path.join(__dirname, global.config.media_path, this.subPath);
}
createFolder(pathName) {
if (pathName.indexOf('..') > -1 || pathName.search(/^\//) > -1)
return false;
let fullPath = path.join(this.basePath, this.subPath, pathName);
let r = fs.mkdirSync(fullPath, { recursive: true });
if (r === undefined) return false;
return true;
}
getFolders() {
let pathName = path.join(this.basePath, this.subPath);
return fs.readdirSync(pathName, { withFileTypes: true })
.filter(f => f.isDirectory())
.map(f => f.name)
}
getFiles() {
let pathName = path.join(this.basePath, this.subPath);
return fs.readdirSync(pathName, { withFileTypes: true })
.filter(f => !f.isDirectory())
.map(f => f.name)
}
}
module.exports = MediaManager;
```
Let's break that down:
```js
const path = require('path');
const fs = require('fs');
```
We'll be merging paths and doing disk operations, so we need the above packages.
```js
class MediaManager {
constructor(pathName=global.config.media_path) {
if (pathName.indexOf('..') === -1 && pathName.search(/^\//) === -1)
this.basePath = path.join(__dirname, pathName);
else
this.basePath = path.join(__dirname, global.config.media_path);
this.subPath = '';
}
```
The `pathName` argument for the constructor is optional; my code has a `global.config` object with a `media_path` property which is a relative path from the website's root folder (where the app script resides). The above code does some basic security steps, making sure the input doesn't attempt to go to the parent directory or start at the root directory.
```js
setSubPath(pathName) {
if (pathName.indexOf('..') > -1 || pathName.search(/^\//) > -1)
return false;
this.subPath = pathName;
return true;
}
```
With the same security steps as the constructor, the `setSubPath()` method, shown above, stores a property so that it can be merged later with the `basePath` property and `__dirname` (this file is in the project root folder, so `__dirname` will return that folder). This is how we 'select' a path with the object.
```js
getPath() {
return path.join(__dirname, global.config.media_path, this.subPath);
}
```
The above is a utility function that gets me the full, absolute path to the currently selected subfolder.
```js
createFolder(pathName) {
if (pathName.indexOf('..') > -1 || pathName.search(/^\//) > -1)
return false;
let fullPath = path.join(this.basePath, this.subPath, pathName);
let r = fs.mkdirSync(fullPath);
if (r === undefined) return false;
return true;
}
```
Again the same security steps. We then create a folder, returning `true` on success and `false` on failure.
```js
getFolders() {
let pathName = path.join(this.basePath, this.subPath);
return fs.readdirSync(pathName, { withFileTypes: true })
.filter(f => f.isDirectory())
.map(f => f.name)
}
```
The above method, `getFolders()`, returns an array of folder names found in the current path selected by the object.
```js
getFiles() {
let pathName = path.join(this.basePath, this.subPath);
return fs.readdirSync(pathName, { withFileTypes: true })
.filter(f => !f.isDirectory())
.map(f => f.name)
}
```
The `getFiles()` method is the same as `getFolders()` (actually identical except for an exclamation mark), but it returns an array of files in the selected folder.
```js
}
module.exports = MediaManager;
```
We end the class and export it.
### Next: multer and the POST Route
First, you need to install `multer` using this command in the terminal:
```
npm install multer
```
Then you need a route that takes advantage of that feature. At the top of the router file (or the top of your main app file, if you aren't using router files), you'll need this:
```js
const multer = require('multer');
const upload = multer({ dest: global.config.media_path });
const MediaManager = require('../media-manager');
```
Note above that my `media-manager.js` file is in the parent directory compared to where my router files sit. Your project structure may be different.
Now we get into the actual route that handles all operations from the media manager:
```js
router.post(
'/media-manager',
upload.single('uploadedFile'),
async (req, res) => {
if (req.session.isAdmin === undefined || !req.session.isAdmin) {
return res.redirect('/user');
}
let m = new MediaManager();
let arg = (req.method === 'POST') ? req.body : req.params;
let r = {};
switch (arg.cmd) {
case 'showDir':
if (m.setSubPath(arg.subPath)) {
r = {
folders: m.getFolders(),
files: m.getFiles(),
}
}
else {
r = { status: false }
}
break;
case 'upload':
r = {};
if (req.file === undefined) {
r = { status: false };
}
else if (!m.setSubPath(arg.subPath)) {
r = { status: false };
}
else {
let file = req.file;
let oldPath = path.join(__dirname, '..', file.path);
let newPath = path.join(m.getPath(), file.originalname);
fs.renameSync(oldPath, newPath, (err) => {
if (err) r = { status: false };
else r = { status: true };
});
}
break;
case 'mkDir':
if (m.setSubPath(arg.subPath) && m.createFolder(arg.newFolder)) {
r = { status: true };
}
else {
r = { status: false };
}
break;
}
res.json(r);
}
);
```
Let's break that down:
```js
router.post(
'/media-manager',
upload.single('uploadedFile'),
async (req, res) => {
```
The above says: if a POST request is made to the `/media-manager` URL, invoke the `upload` middleware to allow a single upload; we're expecting our file to have the form element name `uploadedFile`. And of course, we create a callback function to handle this route.
```js
if (req.session.isAdmin === undefined || !req.session.isAdmin) {
return res.redirect('/user');
}
```
If the user isn't an admin, we redirect them to the login/register page, even if they're already logged in. It strikes me as a good hint ;)
```js
let m = new MediaManager();
```
We create a new MediaManager object called `m`.
```js
let arg = (req.method === 'POST') ? req.body : req.params;
```
Originally this was a `.all` route so I wasn't sure which source the form fields would be found in, `req.body` or `req.params`. The above handles that, moving the form body variables into the `arg` variable.
```js
let r = {};
```
The variable `r` will be our return value.
```js
switch (arg.cmd) {
```
As I said, every action the media manager is capable of, will be handled by this one route. The `arg.cmd` form field (a hidden field, or in some cases just a JavaScript object on the browser side) tells us what operation we're being asked to perform on the server side.
```js
case 'showDir':
if (m.setSubPath(arg.subPath)) {
r = {
folders: m.getFolders(),
files: m.getFiles(),
}
}
else {
r = { status: false }
}
break;
```
If the value of `arg.cmd` is equal to `"showDir"` then we do this code. The `MediaManager` object, `m`, has three methods we're using here. The `m.setSubPath()` method selects which subfolder the user wants to see. (If `arg.subPath` is an empty string, `MediaManager` just stays in the folder it was configured to treat as the root folder.) Then we pack `r`, our return value, with a `folders` field populated by `m.getFolders()` and a `files` field populated by `m.getFiles()`. If for any reason `m.setSubPath()` returns false -- namely security reasons -- then instead the return value becomes an object containing only `status: false`.
```js
case 'upload':
r = {};
if (req.file === undefined) {
r = { status: false };
}
else if (!m.setSubPath(arg.subPath)) {
r = { status: false };
}
else {
let file = req.file;
let oldPath = path.join(__dirname, '..', file.path);
let newPath = path.join(m.getPath(), file.originalname);
fs.renameSync(oldPath, newPath, (err) => {
if (err) r = { status: false };
else r = { status: true };
});
}
break;
```
Thank goodness for that middleware, `multer`. Thanks to it, `req.file` contains our uploaded file data. (Multer can also handle multiple file uploads at once, in which case `req.files` would instead be an array of files rather than `req.file` being a file -- but let's keep it simple for now.) Above, we do two screening steps: if `req.file` is undefined or if we can't select the desired subfolder with `m.setSubPath()` then we set our return value to `{status:false}` and move along home.
But if both of those checks pass, we reach the `else` statement, above, which handles the actual upload. We compose the `oldPath` and `newPath` variables, and use `fs.renameSync()` to move the file from its default location to our desired destination. Notice that, since this router file is in a subfolder of my main project folder, I had to put `'..'` into the `path.join()` call after `__dirname` (which returns the absolute path to the directory where the current file is found). Our return value, `r`, is given a `status` field depending on the success or failure of the `fs.renameSync()` call, and we're done with the file upload process.
```js
case 'mkDir':
if (m.setSubPath(arg.subPath) && m.createFolder(arg.newFolder)) {
r = { status: true };
}
else {
r = { status: false };
}
break;
```
Thanks to my node object `m` (an instance of the `MediaManager` class) the command to make a folder is very simple. Both `m.setSubPath()` and `m.createFolder()` return true or false depending on success, so we can do them both in the `if` statement, connected with `&&` (meaning "logical and", i.e. "if both of these conditions evaluate to `true`"). The functions happen regardless of whether they evaluate to `true` or `false`, except that the 2nd function call (`m.createFolder()`) won't happen if the first one (`m.setSubPath()`) returns false -- a feature called short circuiting.
Oh, and in case you missed it during all that jargon: we've created the folder. If both those functions return true, we set the return value `r` to `{status:true}` and move along home.
```js
}
res.json(r);
}
);
```
This concludes the `switch` and the route handler. We send `r` to the browser with JSON response headers and we're done.
### The Pug Template
This part is easy and requires no explanation if you've used Pug before. Here is `mm.pug` in my `views` folder:
```
div#media_manager
span#close_x X
h3 Folder:
span#folderNameDisplay
div#folder_list
- // populated on client-side
h3 Upload a File
div#upload_form(enctype="multipart/form-data")
#fileuploader Upload
input#subPath(type="hidden", name="subPath", value="")
input#cmd(type="hidden", name="cmd", value="upload")
h3 Create a Folder
form#create_folder
label(for="newFolder") New Folder name:
input#new_folder_name(type="text", name="newFolder", value="")
input#new_folder_submit(type="button", name="submit", value="Create Folder")
```
Of course, the CSS and browser-side JavaScript will be referencing those classes and ID's, so be thorough if you decide to change any of them.
### Pug Code Wherever You Need the Media Manager to Appear
There are two insertions to make in whatever Pug template you want to add the media manager to: a button to make the media manager appear, and an include to invoke the media manager's template.
The button code:
```
div.content_buttons
span.open_mm 🖼️ Media Manager
span.copy_msg Markdown img copied. Now paste it into the content below.
```
The inclusion code:
```
if isAdmin
include mm
script(src="/static/media-manager.js")
```
Every route on my app sends the `isAdmin` variable to the template. Make adjustments for whatever security measure you're using. Also remember indentation is key with Pug templates, so don't blindly assume my indentations in the two snippets above will make sense for your template.
### jQuery and jQuery-uploadfile
In addition to jQuery I'm also using jQuery-UI. Here is the Pug code to include the lot, assuming you have downloaded the packages to your `static` folder:
```
script(src="/static/jquery-ui/external/jquery/jquery.js")
link(rel="stylesheet", href="/static/jquery-ui/jquery-ui.structure.min.css", type="text/css")
link(rel="stylesheet", href="/static/jquery-ui/jquery-ui.min.css", type="text/css")
link(rel="stylesheet", href="/static/jquery-ui/jquery-ui.theme.min.css", type="text/css")
script(src="/static/jquery-ui/jquery-ui.min.js")
script(src="/static/jquery.uploadfile.min.js")
```
### The CSS
I'm not prepared to teach you CSS today; I'll just include the CSS code you need, and encourage you to search W3Schools for the things you're unfamiliar with. Learning how to code is superior to copy-pasting blindly.
```css
/* Media Manager ************************************************************ */
#media_manager {
height: 97vh;
width: 97vw;
position: fixed;
top: 1.5vw;
left: .5vh;
z-index: 4;
background-color: #222;
display: none;
border-radius: 1em;
border: 5px solid #f77a04;
box-sizing: content-box;
}
#media_manager h3 {
margin-left: 10vw;
}
#folderNameDisplay {
margin-left: 25px;
}
.open_mm, #close_x {
cursor: pointer;
}
#media_manager #close_x {
position: absolute;
color: red;
background-color: black;
right: 1em;
top: 1em;
width: 50px;
height: 50px;
font-size: 40px;
font-weight: bold;
padding: 5px;
text-align: center;
}
#media_manager #close_x:hover {
background: red;
color: black;
}
#media_manager #folder_list {
width: 66vw;
height: 55vh;
margin: auto;
padding: 10px;
background-color: #111;
color: #dcdcdc;
border: 1px solid #666;
overflow-y: scroll;
}
#media_manager #folder_list::-webkit-scrollbar,
#media_manager #upload_form::-webkit-scrollbar {
width: 12px; /* width of the entire scrollbar */
}
#media_manager #folder_list::-webkit-scrollbar-track,
#media_manager #upload_form::-webkit-scrollbar-track {
background: #222; /* color of the tracking area */
}
#media_manager #folder_list::-webkit-scrollbar-thumb,
#media_manager #upload_form::-webkit-scrollbar-thumb {
background-color: #444; /* color of the scroll thumb */
border-radius: 20px; /* roundness of the scroll thumb */
border: 3px solid #444; /* creates padding around scroll thumb */
}
#media_manager #upload_form {
width: 66vw;
height: 10vh;
margin: auto;
padding: 10px;
border: 1px solid #666;
}
#media_manager #create_folder {
width: 66vw;
padding: 10px;
margin: auto;
}
.mm_icon_form {
display: inline-block;
vertical-align: top;
margin: 10px 10px;
}
.mm_icon_image {
width: 64px;
height: 64px;
background-color: transparent;
border: 1px solid #333;
font-size: 2em;
}
.mm_icon_form > p {
width: 50px;
font-size: 11pt;
vertical-align: top;
text-align: center;
margin: 0 0;
word-wrap: break-word;
}
.copy_msg {
margin-left: 10px;
display: none;
font-weight: bold;
}
#upload_form {
overflow-y: scroll;
}
#fileuploader {
float: left;
display: inline-block;
}
.ajax-file-upload-statusbar {
border: none;
}
#new_folder_name {
background-color: transparent;
color: #dcdcdc;
padding: 10px;
border: 1px solid #666;
width: 40vw;
margin: auto 15px;
}
#new_folder_submit {
border-radius: 3px;
border: 0px solid white;
}
#upload_form #fileuploader .ajax-file-upload, #new_folder_submit {
color: #dcdcdc;
background-color: rgb(24, 69, 93);
font-family: 'Roboto Condensed';
font-size: 13pt;
padding: 10px;
font-weight: bold;
line-height: 13pt !important;
height: auto;
cursor: pointer;
}
#upload_form #fileuploader .ajax-file-upload form input[type="file"] {
height: 100%;
}
.ajax-file-upload-container {
margin-top: 0;
}
.ajax-file-upload-statusbar {
margin-top: 0;
padding-top: 0;
}
```
### Finally, the Browser-Side JavaScript
```js
var dirContents = {};
function renderFolderContents() {
let folders = files = [];
$('#folder_list').html('');
if (dirContents.folders) folders = dirContents.folders;
if (dirContents.files) files = dirContents.files;
// if subfolder exists make ".." link
let subPath = $('#subPath').val();
let backPath = '';
if (subPath !== '') {
let subArray = subPath.split('/');
if (subArray.length > 1) {
backPath = subArray.splice(subArray.length-1).join('/');
}
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('name', 'back');
i.setAttribute('value', '⬅️');
i.setAttribute('id', '..');
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = 'Go back';
f.appendChild(p);
$('#folder_list').append(f);
document.getElementById('..').addEventListener('click', (e) => {
console.log(e.target.id);
$('#folderNameDisplay').html(
(backPath === '') ? 'Root folder' : backPath
);
$('#subPath').val(backPath);
$.post(
'/media-manager',
{
cmd: 'showDir',
subPath: $('#subPath').val(),
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
}).fail(() => {
})
.always(() => {
});
e.preventDefault();
});
}
folders.forEach((item) => {
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('name', item);
i.setAttribute('value', '📁');
i.setAttribute('id', item);
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = item;
f.appendChild(p);
$('#folder_list').append(f);
document.getElementById(item).addEventListener('click', (e) => {
console.log(e.target.id);
let val = (subPath !== '')
? `${subPath}/${e.target.id}`
: e.target.id;
$('#folderNameDisplay').html(val);
$('#subPath').val(val);
$.post(
'/media-manager',
{
cmd: 'showDir',
subPath: $('#subPath').val(),
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
}).fail(() => {
})
.always(() => {
});
e.preventDefault();
})
});
files.forEach((item) => {
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('value', '');
let sub = ($('#subPath').val()) ? `${$('#subPath').val()}/` : '';
let url = `/static/img/mm/${sub}${item}`;
i.setAttribute('id', item);
i.style.backgroundImage = `url('${url}')`;
i.style.backgroundSize = 'cover';
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = item;
f.appendChild(p);
$('#folder_list').append(f);
document.getElementById(item).addEventListener('click', (e) => {
console.log(e.target.id);
navigator.clipboard.writeText(
``
);
$('#media_manager').fadeOut(150);
$('.copy_msg').fadeIn(1000).fadeOut(5000);
e.preventDefault();
})
});
}
$(document).ready(async () => {
// Media manager always starts at the root folder
$('#folderNameDisplay').html('Root folder');
// initialize jquery-uploadfile plugin
$("#fileuploader").uploadFile({
url: "/media-manager",
fileName: 'uploadedFile',
multiple: false,
dragDrop: false,
dynamicFormData: () => {
return { cmd: $('#cmd').val(), subPath: $('#subPath').val(), };
},
returnType: 'json',
onSuccess: (files,data,xhr,pd) => {
console.log(JSON.stringify(data));
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
},
});
// setup create folder button
document.getElementById('create_folder')
.addEventListener('submit', (e) => {
let newFolder = $('#new_folder_name').val();
if (
newFolder === ''
|| newFolder.indexOf('/') > -1
|| newFolder.indexOf('\\') > -1
) return false;
$.post(
`/media-manager`,
{
cmd: 'mkDir', subPath: $('#subPath').val(),
newFolder: newFolder,
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
console.log(data);
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
})
.fail(() => {
})
.always(() => {
});
e.preventDefault();
});
// get initial folder data
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
$('#close_x').click(() => {
$('#media_manager').fadeOut(150);
});
$('.open_mm').click(() => {
$('#media_manager').fadeIn(150);
});
});
```
You may notice some blank space and think to yourself, that's unlike Astro. Most of it is blanks you can fill in with error-trapping code and other trigger code. I felt I'd be remiss if I removed them, since you might not know the jQuery API supports easily chaining all those options off a single AJAX call.
Let's break it down:
```js
var dirContents = {};
```
Every time we make an AJAX call to refresh the contents of a folder, that data goes in `dirContents` before we call `renderFolderContents()`.
```js
function renderFolderContents() {
```
This is a fairly long function, but also fairly simple.
```js
let folders = files = [];
```
Initialize two variables we'll be looping through later.
```js
$('#folder_list').html('');
```
The above resets the contents of the HTML/DOM element with the id `folder_list`, to blank.
```js
if (dirContents.folders) folders = dirContents.folders;
if (dirContents.files) files = dirContents.files;
```
The `dirContents` object should contain a `folders` field and a `files` field before we hit this function. We're really just making shortcut variables here.
```js
// if we're in a subfolder, make ".." link
let subPath = $('#subPath').val();
let backPath = '';
if (subPath !== '') {
let subArray = subPath.split('/');
if (subArray.length > 1) {
backPath = subArray.splice(subArray.length-1).join('/');
}
```
If we're in a subfolder, we need the ability to navigate to the parent folder. The above is prep work for that UI element. If the `subPath` variable is not an empty string, then we create a temporary array by splitting `subPath` with the forward slash as a delimiter. If that array has more than one element, we `splice` off the last element and then rejoin them using forward slashes, to create the new value of the `backPath` variable.
```js
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('name', 'back');
i.setAttribute('value', '⬅️');
i.setAttribute('id', '..');
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = 'Go back';
f.appendChild(p);
$('#folder_list').append(f);
```
The above looks like a lot because it's so many lines of code, but really we're just creating and customizing HTML/DOM elements and strapping them on the existing web page. We make a `form`, with an `input type="button"` inside, and a `p` below that.
```js
document.getElementById('..').addEventListener('click', (e) => {
console.log(e.target.id);
$('#folderNameDisplay').html(
(backPath === '') ? 'Root folder' : backPath
);
$('#subPath').val(backPath);
```
This is our click listener for the back button. We conditionally set the `innerHTML` of the element with ID `folderNameDisplay` to either the backPath (if it's not an empty string), or the words 'Root folder' if `backPath` was an empty string. We also set the value of an `input type="hidden"` element with ID `subPath` where we keep track of the currently-viewed folder. Now comes the actual AJAX call:
```js
$.post(
'/media-manager',
{
cmd: 'showDir',
subPath: $('#subPath').val(),
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
}).fail(() => {
})
.always(() => {
});
e.preventDefault();
});
```
The above executes our AJAX call to show us the updated contents of the folder. The first argument to `$.post()` is the URL, the second argument is the data we need to send to that URL, the third argument is a success function, and the fourth argument notifies jQuery that we expect the response to be JSON data. After that I left blanks for error handling (the `.fail()` trigger) and an after-process trigger that will always happen (the `.always()` trigger). Finally: because this is AJAX; because the back button is a submit button; and because we don't want the page to refresh when you click it, we use `e.preventDefault()` to prevent the form from submitting the old-fashioned way.
```js
}
```
We're done with the `if` statement, `if (subPath !== '')`.
```js
folders.forEach((item) => {
```
It's time to loop through the `folders` array and render each folder. If you paid attention above, most of the below should look familiar.
```js
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('name', item);
i.setAttribute('value', '📁');
i.setAttribute('id', item);
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = item;
f.appendChild(p);
$('#folder_list').append(f);
document.getElementById(item).addEventListener('click', (e) => {
console.log(e.target.id);
let val = (subPath !== '')
? `${subPath}/${e.target.id}`
: e.target.id;
$('#folderNameDisplay').html(val);
$('#subPath').val(val);
$.post(
'/media-manager',
{
cmd: 'showDir',
subPath: $('#subPath').val(),
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
}).fail(() => {
})
.always(() => {
});
e.preventDefault();
})
});
```
As shown above, the inside of the `folders.forEach` loop is almost exactly the same code as the code for the "back" button. We:
- create some elements, strap them together, and then strap them onto the web page
- create an event listener for clicking on the submit button which represents the folder
- do the AJAX call in that event listener, and prevent traditional form submission from occuring
Things will, once again, be very familiar when we loop the `files` array, below, but with a slight twist:
```js
files.forEach((item) => {
let f = document.createElement('form');
f.className = 'mm_icon_form';
let i = document.createElement('input');
i.className = 'mm_icon_image';
i.style.cursor = "pointer";
i.setAttribute('type', 'submit');
i.setAttribute('value', '');
let sub = ($('#subPath').val()) ? `${$('#subPath').val()}/` : '';
i.setAttribute('id', item);
```
Here comes the twist:
```js
let url = `/static/img/mm/${sub}${item}`;
i.style.backgroundImage = `url('${url}')`;
i.style.backgroundSize = 'cover';
```
Instead of an emoji of a folder icon, we're using the actual image itself as the CSS `background-image` attribute, represented in DOM as the `.style.backgroundImage` property; and setting the CSS `background-size` attribute to `cover` via the DOM attribute `.style.backgroundSize`. With very few differences, everything else is the same until you get to the click handler.
```js
f.appendChild(i);
let p = document.createElement('p');
p.innerHTML = item;
f.appendChild(p);
$('#folder_list').append(f);
```
The click hander starts the same, but...
```js
document.getElementById(item).addEventListener('click', (e) => {
console.log(e.target.id);
```
...then comes this part:
```js
navigator.clipboard.writeText(
``
);
$('#media_manager').fadeOut(150);
$('.copy_msg').fadeIn(1000).fadeOut(5000);
```
Above, we're copying a string to the user's clipboard. This is the same as if they had selected that text on a page and pressed Ctrl-C. We also fade out the media manager, fade in a message, and fade out that message again, slowly. (The message explains that they can now paste the Markdown code.)
```js
e.preventDefault();
})
});
}
```
We prevent the form from submitting the non-AJAX way, end the click handler, end the `files.forEach()`, and end the function.
Now we do our page setup. The following code runs as soon as the browser is ready to render the page:
```js
$(document).ready(async () => {
// Media manager always starts at the root folder
$('#folderNameDisplay').html('Root folder');
```
Above, we set an H3 above the folder viewer to say "Root folder" because that's where the media manager starts.
```js
// initialize jquery-uploadfile plugin
$("#fileuploader").uploadFile({
url: "/media-manager",
fileName: 'uploadedFile',
multiple: false,
dragDrop: false,
dynamicFormData: () => {
return { cmd: $('#cmd').val(), subPath: $('#subPath').val(), };
},
returnType: 'json',
onSuccess: (files,data,xhr,pd) => {
console.log(JSON.stringify(data));
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
},
});
```
The above code initializes the jQuery-uploadfile plugin. We use the `dynamicFormData` field to allow us to feed the AJAX call up-to-date data from our `subPath` field. The `$.post()` call in its `onSuccess` field should be familiar by now.
```js
// setup create folder button
document.getElementById('create_folder')
.addEventListener('submit', (e) => {
```
Now we're going to add an event listener for the `submit` event on the `create_folder` form.
```js
let newFolder = $('#new_folder_name').val();
```
We grab the value of the text input box where the user types the name of the folder they want to create.
```js
if (
newFolder === ''
|| newFolder.indexOf('/') > -1
|| newFolder.indexOf('\\') > -1
) return false;
```
We do some basic security: no slashes allowed.
```js
$.post(
`/media-manager`,
{
cmd: 'mkDir', subPath: $('#subPath').val(),
newFolder: newFolder,
},
(data) => {
console.log(data);
},
'json'
).done((data) => {
console.log(data);
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
```
First we send the `mkDir` value in the `cmd` field of the POST request, along with the `subPath` and `newFolder` fields. Then, when that request reports done, we refresh the folder view.
```js
.fail(() => {
})
.always(() => {
});
})
.fail(() => {
})
.always(() => {
});
```
As before, the above are just blanks you can fill out if you want to.
```js
e.preventDefault();
```
Prevent the form from submitting the non-AJAX way.
```js
});
```
Close out the click handler.
Next we populate the media manager with a view of the root folder (that is, the folder it's configured to treat as its `basePath`). It's the same as the other "get data/render the folder list" chains.
```js
// get initial folder data
$.post(
`/media-manager`,
{ cmd: 'showDir', subPath: $('#subPath').val() },
(data) => {
console.log(data);
},
'json'
).done((data) => {
dirContents = data;
renderFolderContents();
})
.fail(() => {
})
.always(() => {
});
```
Next we set up the close and open buttons to do their things:
```js
$('#close_x').click(() => {
$('#media_manager').fadeOut(150);
});
$('.open_mm').click(() => {
$('#media_manager').fadeIn(150);
});
```
And we're done with the script that runs when the page is ready for display!
```js
});
```
### Reminders
Do not expose this to random users. It's not secure enough. I wrote this in a morning from scratch. Make sure you only allow competent admins to use this! But with that said, I think it's attractive, fast, and gets the job done.
How to Convert a String to Unicode "\u0XXX" Symbols, but Only the Special Characters, using JavaScript
By AstroMacGuffin dated Mon Jul 25 2022 14:31:51 GMT+0000 (Coordinated Universal Time) last updated Sun Jul 31 2022 19:41:51 GMT+0000 (Coordinated Universal Time)The problem: You have a string that needs to be converted to Unicode symbols, *but only the special characters.* You want to leave the spaces and punctuation marks alone. Solution within!
Featured Article
How to Code Relevance-Weighted Search with Node & MongoDB
By AstroMacGuffin dated Sat Jul 23 2022 19:14:09 GMT+0000 (Coordinated Universal Time) last updated Thu Aug 18 2022 14:17:35 GMT+0000 (Coordinated Universal Time) In this post I teach you how to implement a search that sorts results by relevance. Search relevance is calculated as the total number of times each search keyword shows up in the article, added together. To accomplish this, you'll create a search index whenever the article is saved.
Regular expressions will be used to match the keywords against the mongo database, so the search keyword "search" will match the database keyword "searching", but the search keyword "searching" won't match the database keyword "search".
### Ingredients
- some kind of CRUD app already written for articles using Node & MongoDB
- an `articles_search` collection for storing strength of keyword relevance per word per article
- a method for removing punctuation and special characters from the content you want to index
- database methods for deleting and inserting into `articles_search`
- a mongo query that matches search terms against a field in `articles_search`
- a loop to add up total keyword strength per article
- a sorting algorithm to sort articles by total keyword strength
Whenever you add or update an article:
- delete all keyword entries for that article
- parse the contents you want to index from scratch; delete old entries for that article from `articles_search`, then add new
- update the article content normally
For the search, it's actually two find operations (probably because I haven't looked into how MongoDB handles join operations yet). First you get the entries from `articles_search` so you can have the `articles._ids`; second you get the article content.
Some of my code to get you through the relatively hard parts:
### First, a Useful Utility Method
```js
_renderSearchTerms(theString) {
return theString
.replace(/\(/gi, ' ')
.replace(/\)/gi, ' ')
.replace(/\./gi, ' ')
.replace(/!/gi, ' ')
.replace(/@/gi, ' ')
.replace(/#/gi, ' ')
.replace(/\$/gi, ' ')
.replace(/\%/gi, ' ')
.replace(/\^/gi, ' ')
.replace(/&/gi, ' ')
.replace(/\*/gi, ' ')
.replace(/-/gi, ' ')
.replace(/_/gi, ' ')
.replace(/=/gi, ' ')
.replace(/\+/gi, ' ')
.replace(/\{/gi, ' ')
.replace(/\[/gi, ' ')
.replace(/\}/gi, ' ')
.replace(/\]/gi, ' ')
.replace(/:/gi, ' ')
.replace(/;/gi, ' ')
.replace(/"/gi, ' ')
.replace(/'/gi, ' ')
.replace(/`/gi, ' ')
.replace(/,/gi, ' ')
.replace(/>/gi, ' ')
.replace(/</gi, ' ')
.replace(/\//gi, ' ')
.replace(/\?/gi, ' ')
.replace(/\|/gi, ' ')
.replace(/\\/gi, ' ')
.replace(/\n/gi, ' ')
.split(' ');
}
```
The above utility method:
- removes all the unwanted characters from a string you intend to index for search, replacing them with spaces
- splits the result into an array, using a space as the split delimeter
- returns it
### Whenever you Add or Update an Article, Set the Keywords Info in `articles_search`
Next is the method that sets the `articles_search` entries for an article.
```js
async setSearchTerms(arg) {
const d = mdb.db("amgdotcom");
const t = d.collection("articles_search");
let id = (arg._id !== undefined) ? arg._id : arg.id;
if (id === undefined) return false;
await this.deleteSearchTerms(id);
```
We don't want duplicate entries, so we've deleted all the old search terms for this article.
```js
let terms = [
...this._renderSearchTerms(arg.title),
...this._renderSearchTerms(arg.content)
];
```
This creates a merged array; note that there may very well be words in the `title` that are also in the `content` ...that's fine, because right now the `terms` array contains the entire article as an array broken into words -- no sums yet, just a raw list of words exactly as they appeared in the original article. Not until we...
```js
let termCount = {};
terms.forEach((item) => {
item = item.trim().toLowerCase();
if (item !== '') {
if (termCount[item] === undefined) termCount[item] = 1;
else termCount[item]++;
}
});
```
*Now* we have the sums. Next we simply collate a `docs` array full of objects compatible with MongoDB's node driver method, `[db].[collection].insertMany()`.
```js
let docs = [];
// rewriting terms var
terms = Object.keys(termCount);
terms.forEach((item) => {
docs.push({
article_id: ObjectId(id),
term: item,
count: termCount[item]
});
});
```
Finally we insert the `articles_search` entries into the collection (`t` was defined all the way on the 2nd line of this function).
```js
t.insertMany(docs);
}
```
### And Finally, the Search Method
```js
async searchArticles(term, type='blog') {
```
The `term` parameter needs to be a space-separated string of keywords.
```js
try {
let r = [];
let article_weights = {};
const d = mdb.db("amgdotcom");
let t = d.collection("articles_search");
```
Just setting up variables to be used later. The collection `t` will be reassigned later, so it's not a `const` like it otherwise would be.
```js
let query = { $or: [] };
let terms = term.split(' ');
for (let i = 0; i < terms.length; i++) {
query['$or'].push({term: new RegExp(terms[i], 'gi')});
}
```
We're dynamically building a `query` object, above.
```js
let options = {
sort: { count: -1 },
projection: { article_id: 1, count: 1 },
};
```
All we need from the `articles_search` collection is the `articles._id` value stored in `articles_search.article_id`, and the `count` of how relevant the keyword was to that article. We don't care about the keyword, but we do sort by `count`, descending.
```js
const cursor1 = await t.find(query, options);
await cursor1.forEach((item) => {
if (!article_weights.hasOwnProperty(item.article_id))
article_weights[item.article_id] = { count: item.count };
else
article_weights[item.article_id].count += item.count;
});
```
Above, we're priming the `article_weights` object now, calculating the total relevance. The goal is to know how relevant the article is when you take all keywords in the search term into account at once.
```js
// resetting query
if (type === 'all')
query = { $or: [] };
else
query = { type: type, $or: [] };
let article_ids = Object.keys(article_weights);
for (let i = 0; i < article_ids.length; i++) {
query['$or'].push({_id: ObjectId(article_ids[i])});
}
```
Once again, we're building a dynamic query above. This time we want every article that's relevant at all.
```js
// resetting t
t = d.collection('articles');
// resetting options
options = {
projection: {
_id: 1, title: 1, content: 1, tags: 1, username: 1,
creation: 1, updated: 1, type: 1,
},
};
```
Now we're preparing to do a `.find()` on the `articles` collection. I didn't bother sorting on the lazy assumption that I have to do that manually. (Seriously, I should just look up how MongoDB joins work, if at all.)
```js
const cursor2 = t.find(query, options);
```
I love the MongoDB Node.js driver so far.
```js
await cursor2.forEach((item) => {
r.push({
id: item._id,
title: item.title,
content: item.content,
tags: item.tags,
username: item.username,
creation: item.creation,
updated: item.updated,
type: item.type,
});
});
```
The `r` array is the return value. Here we're packing it full of everything needed to render each article that was relevant to the search.
```js
r.sort((a, b) => {
if (article_weights[a.id].count > article_weights[b.id].count)
return -1;
else if (article_weights[a.id].count === article_weights[b.id].count)
return 0;
else return 1;
});
return r;
```
Above, we sort the return value so that the more total keyword matches for an article, the earlier in the array it'll be sorted.
```js
}
catch (e) {
console.log(`There was an error: ${e}`);
}
}
```
The other methods are easy, basic CRUD stuff. Have fun with your fancy new relevance-weighted search!
