Version 1 - History - URL patterns - Haketilo - Hydrilla issue tracker

1

jahoti

# URL patterns

2

3

We want to be able to apply different rules and custom scripts for different websites. However, merely specifying "do this for `https://example.com`" is not enough. Single site's pages might differ strongly and require different custom scripts to be loaded. However, always matching against a full URL like `https://example.com/something/somethingelse` doesn't allow us to properly handle a site that serves similar pages for multiple values substituted for `somethingelse`.

4

5

{{toc}}

6

7

## Currently employed solution

8

Wildcards are being used to address the problem. Each page entry in Hachette settings has a URL pattern that specifies to which internet pages it applies. A URL pattern can be as as simple as literal URL in which case it only matches itself. It can also contain wildcards in the form of one or more asterisks (`*`) that correspond to multiple possible strings occurring in that place.

9

10

Wildcards can appear in URL's domain and path that follows it. These 2 types of wildcards are handled separately.

11

12

### Domain wildcards

13

A domain wildcard takes the form of one, two or three asterisks occurring in place of a single domain name segment at the beginning (left). Depending on the number of asterisks, the meaning is as follows:

14

* no asterisks (e.g. `example.com`) - match domain name exactly (e.g. `example.com`)

15

* one asterisk (e.g. `*.example.com`) - match all domains resulting from substituting `*` with a **single** segment (e.g. `banana.example.com` or `pineapple.example.com` but **not** `pineapple.pen.example.com` nor `example.com`)

16

* two asterisks (e.g. `**.example.com`) - match all domains resulting from substituting `**` with **two or more** segments (e.g. `monad.breakfast.example.com` or `pure.monad.breakfast.example.com` but **not** `cabalhell.example.com` nor `example.com`)

17

* three asterisks (e.g. `***.example.com`) - match all domains resulting from substituting `***` with **zero or more** segments (e.g. `hello.parkmeter.example.com` or `iliketrains.example.com` or `example.com`)

18

19

### Path wildcards

20

A path wildcard takes the form of one, two or three asterisks occurring in place of a single path segment at the end of path (right). Depending on the number of asterisks, the meaning is as follows:

21

* no asterisks, zero or one trailing slashes (e.g. `/joke/clowns`) - match path exactly (e.g. `/joke/clowns`) or with arbitrary number of trailing slashes added (e.g. `/joke/clowns/` or `/joke/clowns//`)

22

* one asterisk (e.g. `/itscalled/*`) - match all paths resulting from substituting `*` with a **single** segment (e.g. `/itscalled/gnulinux` or `/itscalled/glamp` but **not** `/itscalled/` nor `/itscalled/gnu/linux`), possibly with additional trailing dashes added (e.g. `/itscalled/glamp/` or `/itscalled/glamp//`)

23

* two asterisks (e.g. `/another/**`) - match all paths resulting from substituting `**` with **two or more** segments (e.g. `/another/nsa/backdoor` or `/another/best/programming/language` but **not** `/another/apibreak` nor `/another`), possibly with additional trailing dashes added (e.g. `/another/nsa/backdoor/` or `/another/nsa/backdoor//`)

24

* three asterisks (e.g. `/dmarc/***`) - match all paths resulting from substituting `***` with **one or more** segments (e.g. `/dmarc/spf` or `/dmarc/dkim/failure` but **not** `/dmarc`), possibly with additional trailing dashes added (e.g. `/dmarc/spf/` or `/dmarc/spf//`)

25

26

> Why did I not make three asterisks semantically equivalent to their

27

> domain name version? I don't know. Perhaps I had some good reason I

28

> forgot?

29

30

Additionally, any path with literal trailing asterisks is matched by itself, even if such pattern would otherwise be treated as wildcard (e.g. `/gobacktoxul/**` matches `/gobacktoxul/**`). This is likely to change in the future and would best not be relied upon. Appending three additional asterisks to path pattern to represent literal asterisks is being considered.

31

32

### Future additions

33

Right now a URL's query string is being completely disregarded for the purpose of matching the URL patterns. In the future, support for matching URLs with specific query parameters and maybe even HTTP(s) requests with specific POST parameters or specific cookies might me added.

34

35

Currently, protocols in the URL are matched exactly. Making protocol wildcard might make little sense because sites are very unlikely to be serving similar content over for example `http://` and `ftp://`. However, support for some wildcard protocol that matches both `http://` and `https://` might be added in the future.

36

37

The wildcards that have been added so far were designed to allow a reasonable level of flexibility, considering some common ways websites are served. However, only practice can show what works best, and so wildcard semantics are subject to change as the project matures.

38

39

### Wildcard priorities and querying

40

Code that handles wildcards resides in `background/settings_query.js`.

41

42

Currently, when querying settings for a URL, possible patterns matching it are computed in order and if a setting for one of the patterns exists, it is returned. This is a temporary mechanism that is later going to be replaced with a more optimal one (although it is not as tragically slow as it might seem).

43

44

In case multiple patterns match some URL, the more specific one is preferred. Specificity is considered as follows:

45

* If patterns only differ in the final path segment, the one with least wildcard asterisks in that segment if preferred.

46

* If patterns, besides the above, only differ in path length, one with longer path is preferred. Neither final wildcard segment nor trailing dashes account for path length.

47

* If patterns, besides the above, only differ in the initial domain segment, one with least wildcard asterisks in that segment is preferred.

48

* If patterns, differ in domain length, one with longer domain is preferred. Initial wildcard segment does not account for domain length.

49

50

As an example, consider the URL `http://settings.query.example.com/google/tries/destroy/adblockers//`. Patterns matching it would be tried in the following order:

51

52

```

53

http://settings.query.example.com/google/tries/destroy/adblockers/

54

http://settings.query.example.com/google/tries/destroy/adblockers

55

http://settings.query.example.com/google/tries/destroy/adblockers/***

56

http://settings.query.example.com/google/tries/destroy/*

57

http://settings.query.example.com/google/tries/destroy/**

58

http://settings.query.example.com/google/tries/destroy/***

59

http://settings.query.example.com/google/tries/**

60

http://settings.query.example.com/google/tries/***

61

http://settings.query.example.com/google/**

62

http://settings.query.example.com/google/***

63

http://settings.query.example.com/**

64

http://settings.query.example.com/***

65

http://***.settings.query.example.com/google/tries/destroy/adblockers/

66

http://***.settings.query.example.com/google/tries/destroy/adblockers

67

http://***.settings.query.example.com/google/tries/destroy/adblockers/***

68

http://***.settings.query.example.com/google/tries/destroy/*

69

http://***.settings.query.example.com/google/tries/destroy/**

70

http://***.settings.query.example.com/google/tries/destroy/***

71

http://***.settings.query.example.com/google/tries/**

72

http://***.settings.query.example.com/google/tries/***

73

http://***.settings.query.example.com/google/**

74

http://***.settings.query.example.com/google/***

75

http://***.settings.query.example.com/**

76

http://***.settings.query.example.com/***

77

http://*.query.example.com/google/tries/destroy/adblockers/

78

http://*.query.example.com/google/tries/destroy/adblockers

79

http://*.query.example.com/google/tries/destroy/adblockers/***

80

http://*.query.example.com/google/tries/destroy/*

81

http://*.query.example.com/google/tries/destroy/**

82

http://*.query.example.com/google/tries/destroy/***

83

http://*.query.example.com/google/tries/**

84

http://*.query.example.com/google/tries/***

85

http://*.query.example.com/google/**

86

http://*.query.example.com/google/***

87

http://*.query.example.com/**

88

http://*.query.example.com/***

89

http://**.query.example.com/google/tries/destroy/adblockers/

90

http://**.query.example.com/google/tries/destroy/adblockers

91

http://**.query.example.com/google/tries/destroy/adblockers/***

92

http://**.query.example.com/google/tries/destroy/*

93

http://**.query.example.com/google/tries/destroy/**

94

http://**.query.example.com/google/tries/destroy/***

95

http://**.query.example.com/google/tries/**

96

http://**.query.example.com/google/tries/***

97

http://**.query.example.com/google/**

98

http://**.query.example.com/google/***

99

http://**.query.example.com/**

100

http://**.query.example.com/***

101

http://***.query.example.com/google/tries/destroy/adblockers/

102

http://***.query.example.com/google/tries/destroy/adblockers

103

http://***.query.example.com/google/tries/destroy/adblockers/***

104

http://***.query.example.com/google/tries/destroy/*

105

http://***.query.example.com/google/tries/destroy/**

106

http://***.query.example.com/google/tries/destroy/***

107

http://***.query.example.com/google/tries/**

108

http://***.query.example.com/google/tries/***

109

http://***.query.example.com/google/**

110

http://***.query.example.com/google/***

111

http://***.query.example.com/**

112

http://***.query.example.com/***

113

http://**.example.com/google/tries/destroy/adblockers/

114

http://**.example.com/google/tries/destroy/adblockers

115

http://**.example.com/google/tries/destroy/adblockers/***

116

http://**.example.com/google/tries/destroy/*

117

http://**.example.com/google/tries/destroy/**

118

http://**.example.com/google/tries/destroy/***

119

http://**.example.com/google/tries/**

120

http://**.example.com/google/tries/***

121

http://**.example.com/google/**

122

http://**.example.com/google/***

123

http://**.example.com/**

124

http://**.example.com/***

125

http://***.example.com/google/tries/destroy/adblockers/

126

http://***.example.com/google/tries/destroy/adblockers

127

http://***.example.com/google/tries/destroy/adblockers/***

128

http://***.example.com/google/tries/destroy/*

129

http://***.example.com/google/tries/destroy/**

130

http://***.example.com/google/tries/destroy/***

131

http://***.example.com/google/tries/**

132

http://***.example.com/google/tries/***

133

http://***.example.com/google/**

134

http://***.example.com/google/***

135

http://***.example.com/**

136

http://***.example.com/***

137

http://**.com/google/tries/destroy/adblockers/

138

http://**.com/google/tries/destroy/adblockers

139

http://**.com/google/tries/destroy/adblockers/***

140

http://**.com/google/tries/destroy/*

141

http://**.com/google/tries/destroy/**

142

http://**.com/google/tries/destroy/***

143

http://**.com/google/tries/**

144

http://**.com/google/tries/***

145

http://**.com/google/**

146

http://**.com/google/***

147

http://**.com/**

148

http://**.com/***

149

http://***.com/google/tries/destroy/adblockers/

150

http://***.com/google/tries/destroy/adblockers

151

http://***.com/google/tries/destroy/adblockers/***

152

http://***.com/google/tries/destroy/*

153

http://***.com/google/tries/destroy/**

154

http://***.com/google/tries/destroy/***

155

http://***.com/google/tries/**

156

http://***.com/google/tries/***

157

http://***.com/google/**

158

http://***.com/google/***

159

http://***.com/**

160

http://***.com/***

161

```

162

163

For a simpler URL like `https://example.com` the patterns would be:

164

165

```

166

https://example.com/

167

https://example.com

168

https://example.com/***

169

https://***.example.com/

170

https://***.example.com

171

https://***.example.com/***

172

https://*.com/

173

https://*.com

174

https://*.com/***

175

https://**.com/

176

https://**.com

177

https://**.com/***

178

https://***.com/

179

https://***.com

180

https://***.com/***

181

```

182

183

## Alternative solution: mimicking web server mechanics

184

While wildcard patterns as presented give a lot of flexibility, they are not the only viable approach to specifying what URLs given settings of custom scripts should be applied to. In fact, wildcards are different from how the server side of a typical website decides what to return for a given URL request.

185

186

In a typical scenario, an HTTP server like Apache reads configuration files provided by its administrator and uses various virtual host, redirect, request rewrite, CGI, etc. instructions to decide how to handle given URL. It is possible using a schema that mimics the configuration options typically used with web servers would give more efficiency in specifying what page settings to apply when.

187

188

This approach shall be considered in the future.

Project

General

Profile

Haketilo

URL patterns » History » Version 1